Research methods and marshalling messy data: Dear Diary

2nd September 2015

Just a few months ago, when the sheer scale of my current project was beginning to overwhelm me, I began to keep a research diary. 

I had set out to examine why, after two decades of international intervention and aid, armed groups were not only as prevalent in the DRC as they had ever been, but were proliferating. The question I was trying to answer was admittedly a broad one – what is it that we’ve missed about armed groups? Researching this topic revealed more questions than answers. 

In the early stages the focus of my research changed almost weekly; each new discovery revealed new significant themes. But, having finally decided upon the focus – the dynamics that drive armed-group violence – I found a plethora of tangentially relevant issues which I would have loved to have probed more deeply. I couldn’t, of course, because I needed to focus on the task in hand. 

Similarly, I came across gaps in related bodies of literature, potential comparative case studies and interesting people to collaborate with. These would have all been interesting large-scale projects in their own right, but they were also distractions that led me down what the urban dictionary calls a 'Wiki-hole' of wasted hours as I pursued information online about issues beyond my research focus.

“A research diary would at least give me somewhere to ‘park’ tangential ideas – and I might well find that not all of them had been moments of dead-end procrastination.”

As I struggled to maintain my focus, our team’s communications support officer suggested that I could record my tangential ideas in a research diary. This would at least give me somewhere to park them, useful or not – and, in any case, as time went by and I reviewed my diary entries, I might well find that not all my unfocused ideas had been dead-end moments of procrastination.

So, when I wondered whether exorbitant UN salaries in DRC might provide negative incentives for peacekeepers, writing a diary entry about it helped me overcome the urge to laboriously follow up on UN pay scales, which could have seem me disappear down the wiki-hole for days. 

The diary helped me file away ideas so that I could focus on my current research. But it also helped me realise how much time I spend thinking about my research, even when I’m reading for pleasure. For example, I made a note about how in John Buntin’s L.A. Noir, I was struck by the dynamics he describes between the LAPD and the LA gangs, and how this echoed the evidence I was gathering about state and armed gang interaction in the DRC. 

Then there was the day I read an essay by Howard Jacobson about the opening line from Kafka’s The Trial: ‘Someone must have been telling lies about Joseph K…’. Yet Jacobson comes up with an interesting observation – nowhere in the story is it confirmed that anyone had been telling lies. For me, this prompted a train of thought about rumour and assumption – not all rumours are true, but not all rumours are lies either. In the DRC, it was crucial to remember this and it has since fed into the way I’ve handled my interview data. 

“The diary allows me to record what interests or enrages me, released from self-censoring worries about political correctness or literary polish.”

The diary allows me to write freely, for myself, recording what interests (or enrages) me, and it’s not too time-consuming because I’m released from self-censoring worries about political correctness or literary polish. Yet it has already proved a very useful source of data for an autoethnography of my research experiences, both remotely and in the field, in the eastern DRC, on the use of crowdsourcing in conflict-affected environments. 

I kept field diaries while I was in the DRC but it was in my research diary, kept quite separately, that I found I had a good record of all the difficulties I’d faced getting there. The obstacles I had to overcome to set up the crowdsourcing technology, the problems with the project (and my reaction to them as they unfolded) – all this information was to hand, along with my reflections on the implications for knowledge production in conflict-affected states. Only now do I realise how valuable recording all this has been for my research. 

With hindsight, I can offer the following advice to anyone considering keeping this kind of diary.

  • Discipline yourself to write, daily if possible – the entry need not be lengthy, but just a few sentences will help overcome any fear of writing.
  • Keep it manageable – limit yourself to one reflection a day.
  • Regularly review your entries (weekly is good, with revisits once every few months), otherwise they become a collection of insights that are as lost to you as if you had never written them down. 
  • Keep a contents page to make review quicker – a few trigger words that help you remember what each entry was about.
  • Not all the thoughts you record will be winners. Some lead nowhere, and that’s normal.

In a credit-obsessed academic culture it’s easy to dismiss a research diary as having little impact or REF-value. But for me, at least, it has become a valuable tool that illuminates my research process and has prevented me from becoming overwhelmed by messy data. 


Hear more from Suda in this seminar podcast on research and risk-aversion in conflict-affected societies.


Image: Diary (Photo: KévinV.Pxl, Flickr)


Leave a comment

The views expressed in Opinions posts are those of the author(s) and do not necessarily represent those of DLP, the Australian Government or DLP's partner organisations.




Suda Perera

Suda Perera

Suda is a DLP Research Fellow based at the University of Birmingham's International Development Department. Her doctoral thesis examined the role of Rwandan refugees in the conflict dynamics of the eastern Congo. Suda’s current research focuses on the role of non-state actors in developmental leadership. For example, she is examining how armed groups in the eastern Democratic Republic of the Congo can be transformed into legitimate political actors who provide wider representation for marginalised citizens. 

Read more

Related items

Education, development, and the problem with consensus

Why rethink the international consensus on 'quality basic education for development'?

Opinion by Michele Schweisfurth7th April 2014

Corruption research: Hunting for glimmers of light in the gloom

Why it can be hard to start a conversation with the people who might know what really works in the fight against corruption.

Opinion by Caryn Peiffer25th July 2017

Service delivery and state legitimacy: For better or for worse?

DLP research fellow Claire Mcloughlin challenges the widely held assumption that there is a self-reinforcing 'virtuous circle' between service delivery and state legitimacy. 

Opinion by Claire Mcloughlin24th November 2015

Corruption: unpacking the black box of political will

New thinking on the reasons why individuals engage in corruption - including the pragmatic calculation that, right or wrong, corruption may be the only solution to pressing difficulties. 

Opinion by Heather Marquette12th January 2015

Bringing Political Economy Analysis in from the cold

Once seen as a 'transformative' tool to change donor thinking, does much PEA now do little to help staff think and work politically?

Opinion by Jonathan Fisher6th May 2014

How does politically informed programming shape development outcomes?

A new 'thinking and working politically' community of practice aims to develop practical guidance for development practitioners based on evidence of what works in politically smart programming.

Opinion by Mark Robinson29th January 2016

Being 'there': Reflections on fieldwork in the DRC

Fieldwork in fragile places - Part 1: The security dilemma. Staying safe while collecting the data that matters. 

Opinion by Suda Perera5th November 2014

Masculinity and sexual violence in India

The brutal rape and murder in December 2012 of a 23-year-old student in a Delhi bus has been the catalyst for rapidly evolving activism against sexual violence in India.

Opinion by Martin Rew16th September 2015

How to find surprising development successes

How mixed methods can unearth cases of positive deviance.

Opinion by Caryn Peiffer16th April 2018

Research methods and marshalling messy data: Dear Diary

The useful role a research diary can play in the assimilation and ordering of qualitative data. 

Opinion by Suda Perera2nd September 2015

Beyond perceptions of corruption?

Corruption is hard to define without straying into the subjective. It's also difficult to build robust methodologies to investigate it.

Opinion by Caryn Peiffer5th February 2015

Being 'there': Bermuda Triangulation

Fieldwork in fragile places - Part 2: Data difficulties. Adapting methodology to 'messy' contexts.

Opinion by Suda Perera6th November 2014