Suad Joseph

AUGUST 29, 2022

The methods and approaches used in the MPK project are a work in progress.  The faculty and graduate student researchers have invented and reinvented the methods continually throughout the project.  We engaged with technologies most of us had not used before (many of which are not generally used in the social science and humanities fields in which we work). We consulted with librarians, archivists, STEM colleagues, web designers, and built on ideas from them all. We researched and experimented with types of literature reviews, databases, software, search techniques and tools, mapping, web designs, and myriads of other steps and strategies.  Many were useful, some we abandoned, others we struggled with but continued to use.  None of our methods were set in stone at the beginning, middle, or end. It is a project that will continue to transform as project members learn better methods and approaches and as more effect ones emerge.

The goal of the MPK project has been to identify all the research carried out on women and gender in the Arab region from 1970 to the present under the organizing topic of “socialities.” While the project has as its goal the mapping of knowledge produced under the organizing topics of the political and the economic as well, our funding allowed us to begin with socialities, within which we eventually developed six thematics (See document “Why Thematics”). The project is, in the first instance, descriptive – laying out and mapping the work that has been done:  what was the work, who did it, when did they do it, on what topics did they do it, where did they do it, where did they publish the results, and so forth. In the second instance, the project is analytical – critically evaluating what has been produced within the socio/political historical contexts in which the research was produced in order to lay the foundation for training early career scholars for more pioneering work.

Suad Joseph (University of California, Davis) proposed the project in 2015 to the University of California Davis Arab Region Consortium (UCDAR  http://joseph.ucdavis.edu/ucdar ) Steering Committee. Nawar Al-Hassan Golley (American University of Sharjah), (Natalie Honein Shihadi of AUS joined briefly in 2016) and Martina Rieker (American University in Cairo) were early project members who have stayed the course from the early launch. American University of Beirut partners shifted (Nadia El Cheikh, Charlotte Karam) until Sara Mourad joined in the Fall of 2016. Lebanese American University project members also changed (Lina Abirafeh, Myriam Sfeir, Connie Christiansen, Jennifer Skulte-Ouaiss) until Zina Sawaf joined in 2020.  Birzeit University started with Eileen Kuttab, until Lena Meari joined in 2016). Seteney Shami (Arab Council for the Social Sciences) participated, as did some of ACSS staff (Nada Chaya, Deema Kaedbay, Moushira Elgeziri) for several meetings.

This document lays out the process, the decisions, the steps we, as a team, took to develop the database and the critical analyses of the literature. We call it the “methods” document with full awareness that methods cannot be separated from theory or from politics. We are aware that the steps we took, the decisions we made, procedural or methodological, were shaped by our own theoretical commitments, as well as the structural givens and constraints in the production of knowledge in general, and specific to women and gender in the Arab region. For example, we dedicated several workshops to the larger framing of the project. That we ended up organizing the scholarship under the large rubrics of “socialities,” the “political,” and “the economic,” is not only the result of days and days of discussions, but also of the ways in which scholarship has been organized over the past century, even though we know these fields are interconnected. That we focused on “socialities” first is a result of the funding opportunity that we had. The six thematics we developed under “socialities” is a result of specific interests of our faculty team, as well as the landscape of research. The fact that the overwhelming majority of the databases that exist and were efficient are English language, and or European, is itself the result of the politics of funding and the colonial histories within which production of knowledge is embedded.

We are aware of the impact of scholarly conventions, funding, commercialization of knowledge production, and the politics of knowledge production on the product that we would/could produce. Some of the platforms we used to search for references claimed intellectual property over the abstracts in the RIS files (Research Information Systems). That meant that we could not include abstracts from those platforms in the database, although we could use them for coding the RIS files and we could include abstracts from other platforms. Access to the various platforms we used (Proquest, EBSCO, BASE, etc.) is by subscription. This means that users who do not have access to subscriptions cannot link directly to the articles in those databases. The fact that English language articles are indexed much more extensively than other languages meant that the MPK database is overwhelmingly in English. The fact that indexing scholarship in Arabic is still relatively new and not yet systematic and by no means thorough, meant that we missed important literature in Arabic, despite our committed efforts.

The specific the Boolean Operators we developed to search the databases were less the result of our intellectual innovation and more a bending to the ways scholars have organized their research over the past half century. (Boolean Operators, are the string of key terms connected by phrases and terms such as “and,” “or,” and organized within (   ) and other markers that are the standards used to search for references within databases.)

The Boolean Operators, which informed the code book, are a product of categorical thinking. Categories are products of imagination, social and personal. We are aware not only of the arbitrariness of some categories, but also the historical limitations/constraints of many categories (Joseph 2022). As all knowledge production is historically situated, we are aware that the categories we used to search and design the Code Book for others to search shifted and changed over time. Our goal is to capture that history, trace the genealogy, and render it more accessible for analysis.

We made mistakes. We knew we would make mistakes. We produced an imperfect product. We knew we would produce an imperfect product. The goal was to make a start, to lay a foundation, to build a foundation for future scholars. The purpose of this “methods” document is to track those steps so that other scholars may hopefully use our experience to build better.

The faculty team (Suad Joseph, Sara Mourad, Martina Rieker, Zina Sawaf, Lena Meari, Nawar Al-Hassan Golley) began zoom sessions in Spring 2020 to plan the work, meeting weekly. Initial meetings consisted of reviewing the proposal that had been funded and operationalizing key concepts and steps to be taken. Winter of 2021 was devoted to interviewing and hiring graduate student researchers (GSRs), all of whom were from the UCDAR universities.  Since one of the goals of the MPK project is to mentor and train graduate students and early career scholars, we set the criteria for the GSRs, in general, as fluency in English and Arabic — and French where possible; well-read in and committed to gender studies and Arab region studies; availability for 18 months and ability to work an average of 10 hours a week.

The faculty continued to meet weekly to frame the project.  We focused especially on researching and discussing the different kinds of “reviews” we would consider carrying out for mapping the literature after we had carried out our searches and built our databases: Blasts, Scoping Reviews, Systematic Reviews, Qualitative Systematic Reviews, Critical Systematic Reviews. It became clear that the standard STEM Systematic Review, relying on Covidence and the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) evaluation templates would not serve our purposes. Systematic Reviews are carried out mostly by STEM scholars, especially in the medical fields, when there is a concrete and narrow question to answer. It requires “judges” to read and evaluate the relevance of the literature to the question being searched. The MPK project did not have such a concrete question.  The MPK goal was a large-scale mapping of all the published scholarly research, followed by qualitative critical reviews of the literature.  For this goal, it was more appropriate to do a Scoping Review which is a broad gathering of relevant literature on specific topics.

Following the hiring of the GSRs in Winter of 2021, the faculty and GSRs met together weekly or separately with the PI (Suad Joseph).  The GSR meetings were focused on training in the social science databases; in Proquest, EBSCO, and other platforms; in Zotero; in Boolean Operators; in Covidence; in Box; in developing codes; in building the Code Book; in content analysis; in reading abstracts and coding them.  Each MPK member was sponsored to acquire a University of California, Davis email address in order to access the extensive University of California libraries and database platforms.

For these trainings, we called upon UC Davis librarians Adam Siegel and Bruce Abbott for training in the databases and Boolean Operators. Prof. Patrick Marius Koga, UCD School of Medicine, gave several lectures to GSRs and faculty on Covidence, PRISMA, Systematic Reviews, Boolean Operators, Zotero. In reality, training was on going the full length of time of the project for the GSRs and the faculty. We were continually considering different tools, software, search engines, platforms, storage possibilities, and the like. Often, it was the GSRs who knew or familiarized themselves with the tools and trained the other GSRs or the faculty. In this, GSR Meghan Klasic, who completed her PhD during the MPK project, repeatedly took the lead to train.

We decided it would be important to document our processes, methods, and steps to both understand what we had accomplished as well as offer a template to be shared with other scholars and students. The GSRs were asked to submit weekly “methods” documents along with their weekly reports. Meghan Klasic produced and maintained the collective methods document from 2021 until Summer of 2022, when El Emira Joanna Abillama took charge of the document. We recorded a large percent of our Zoom meetings and posted them on the MPK Box for the team to review. The Methods documents and the weekly reports were all posted to our shared Box folder at UC Davis.

UCD librarian Adam Siegel suggested platforms and databases for the MPK project to consider:

* Web of Science (UCD)
* SCOPUS
International disciplinary indexes

* Sociological Abstracts (UCD)
* Anthropology Plus (UCD)
* ERIC (Education)
* PubMed
* Women’s Studies International (UCD)
* Gender Watch (UCD)
* LGBTQ+ Life (UCD)
European-language indexes

* Bielefeld Academic Search Engine (BASE)
* Cairn Info (France)
* Francis (CRNS)
* Persee (France)
* Gallica (France)
Arabic World / Africa

* AfricaBib
* Shamaa – Arab Educational Information Network
* Access to Scientific and Socio-economic Information in Africa (ASKIA) (Unesco)
* Al-Manhal (AUC)
* AskZad (AUC)
Full-Text Journal Archives
* JSTOR (UCD)
* Project Muse (UCD)
Full-Text Newspaper Archives
Factiva (UCD)
Arabic Indexes
* http://www.dfaj.net/
* http://srv2.eulc.edu.eg/eulc_v5/libraries/start.aspx?ScopeID=1.&

We researched and considered the following databases for references: Sociological Abstracts (data from 6,000+ journals under Proquest management); JSTOR (full text journals, Elsevier management); Anthropology Plus (located at Harvard, under EBSCO management); Project Muse (full text journals, Elsevier management); Women’s Studies International (EBSCO management); Bielefeld Academic Search Engine (BASE management); PsycInfo (exhaustive database for psychological sciences, Proquest management); Web of Science (Clarivate Analytics management); Scopus (Elsevier management); ERIC (maintained by US Department of Education, Proquest management); PubMed (National Library of Medicine management); EconLit (published by American Economic Association); Gender Watch (Proquest management; overlaps with Women’s Studies International, EBSCO management); LGBTQ+Life; Cairn Info (Belgian & French publishers); Francis (similar to Pascal); Persee (French magazines); Gallica (Bibliotec Nacional); Factiva (newspapers).  For Arabic we discussed Shamaa (Arab Educational Information Network); AfricaBib; Al-Manhal (Cairo based); AskZad (commercial/private company); Acess to Scientific and Socio-Economic Information in Africa (ASKIA, UNESCO).

We could not search all the databases. We started with Bieldefeld Academic Search Engine (BASE) and Sociological Abstracts (on Proquest platform) as a site for blast searches, to learn what was available and what we would need to do to access the databases. We decided to develop the protocols for the Arabic searches and databases after completing English and other European languages where most of the scholarship is indexed. As we were later to learn, the program management for the databases is critical, as they produce require different Boolean Operators and search protocols.

We focused on developing the Boolean Operators for the searches for each of the six thematics, under the topic of socialities. Each GSR was assigned one thematic (we started out with six GSRs) to work with one of the faculty mentors. The teams of GSRs/mentors developed the Boolean Operators for their thematics. We carried out blastsearches to test the efficacy of our search protocols and test the Boolean Operators. We used the term blast to describe global searches we did to test the Operators. Each GSR carried out blast searches for their specific thematic. We met weekly to discuss whether the Operators captured the literature we were interested in mapping. We decided on a series of global Operators that would be a part of each search (see Code Book).  These were revised periodically as we experimented. Parallel with the development of the Operators was the development of the Code Book. The Code Book, in effect, is the organization of the Boolean Operators and the rationale for them for each of the six thematics.  Developing the Operators and the Code Book required a year of work as we kept testing and finding the limits of the design.

We were very aware, as we developed the Boolean Operators for the searches and the Code Book for coding the results, that we were doing “categorical” work (see above). Given that all knowledge production is historically situated, we are aware that the categories we used to search and design the Code Book shifted and changed over time. We were not inventing our own categories. Rather we worked to capture the categories that were used, in different historical moments, starting in 1970s, to study and understand women/gender and socialities in the Arab region. Developing the search terms for the Boolean Operators and the Code Book became a critical exploration of the genealogies of knowledge production on women and gender in the Arab region.

Knowing these constraints, we developed criteria for inclusion of references in the MPK database. The scholarship had to be: 1.) about the Arab region (not the diaspora); 2) about women and gender; 3.) if about men, it had to be about their gender and masculinity, not about men as generic stand-ins for socialities; 4.) about socialities; 5.)  emerging from academic journals, book chapters, books, and grey literature from women’s regional NGOs, and UN reports.

Simultaneously, with the training, the blasts, and the explorations, the faculty mentors worked with the UCD librarians to decide on which databases to prioritize. Adam Siegel, UCD librarian, researched the databases appropriate to the MPK project and gave us a list of U.S.-based, Europe-based, and Arab region-based English language, European languages, and Arabic language databases (see list above). We decided to do one database at a time, carrying out all the steps that we planned to engage in with one database in order to sort out any problems. Then we would proceed, hopefully with greater speed to other databases. The decision was to start with Sociological Abstracts, under Proquest program management, as it was considered to be one of the best organized for MPK purposes.

We searched for English, French, and Arabic, but the Sociological Abstracts database is heavily English language. We came to understand that digitalization of European research is not as developed, and opted to include the Bielefeld database (BASE) as the leading database of European social science research. We planned on developing the tools for Arabic searches separately. We understood there are very few digitalized databases in Arabic. Time, funds, and expertise limited our language exploration to these languages.  We decided to include grey literature, since many scholars in the Arab region are located in NGOs and often critical research is produced by NGOs or UN reports.

As we were developing Boolean Operators and the Code Book, we discussed searching and coding: 1.) for the six thematics under the topic of socialities; 2.) for theoretical approaches; 3.) for methodological approaches; 4.) for specific countries as well as for the whole Arab region; 5.) for language of the research; 6.) by publishers (journals, book publishers, etc.).  Since we planned on using only titles, authors, abstracts for the coding, it became clear that it would be difficult to identify the theoretical or methodological approaches from the abstracts.  We decided to not code for theoretical or methodological approaches, but address them in the qualitative critical reviews.  We focused on developing Boolean Operators for the six thematics, the countries, and the Arab region for Sociological Abstracts.  We created a Literature Database Tracker Sheet on Google Docs to track all the searches that had been carried out and their results.

The development and testing of the Boolean Operators took several months. The Code book continued to be developed and redesigned for a full year. The testing of the Boolean Operators and the Code Book initially was done through blast searches. Each GSR carried out blast searches for their specific thematic. The process led to constant refining of the Boolean Operators and the Code Book.  The revisions of the Operators and the Code Book continued for over six months, even as we were finalizing the searches from Sociological Abstracts.

We created a Code Book with three columns, eventually. The left-hand column was the term used for coding in Zotero. The middle column was the Boolean Operators used to carry out that specific search. The right-hand column offered the explanation of the terms and operators. We developed a manual for the GSRs on how to search/save/delete/code in Zotero. Later we developed color codes for the key terms that were added when we designed the algorithm for autocoding (see below) that had not been searched through Proquest or EBSCO, etc, but would be still productive for users of the MPK database. We constantly revised the Code Book, to correct discrepancies, add key terms, or edit typos.

When we thought we had the Boolean Operators and Code Book developed, we began the searches of Sociological Abstracts. We immediately found problems with Zotero. The initial searches landed each GSR with over 100,000 references for each thematic. Six GSRs, each with that load, crashed Zotero numerous times, despite the fact that we had paid for extra storage on Zotero.  It was clear that Zotero was not designed for the load we produced, as extensive conversations on the Zotero chat revealed. Working with the online version versus the desktop version of Zotero also made for differences in how much time was required for downloads. For GSRs located in Beirut (four out of six), there was a constant problem of internet access, electrical power outages, and the like. In addition, if several of the GSRs were working at the same time on Zotero, the system crashed.  The “Tags” that came along with Zotero presented a problem, for our coding.  It was too time consuming to delete all the tags built into Zotero for each RIS file (Research Information Systems).  The decision was to put “MPK” at the beginning of each code so that our codes could easily stand out from the Zotero Tags. Zotero Tags were deleted at the end of the search. The loss of one GSR meant that another GSR took on two thematics to search and had to redo some searches from the beginning. The loss of a second GSR, later, meant that the four GSRs took on a bigger load of work and again redo some work.

The storage pressure on Zotero was partly because many of the references were not relevant or were duplicates. Removing irrelevant references and duplicates consumed enormous amount of GSR time as they could only be removed by reading each abstract one by one. We were all aware of the arbitrariness of separating socialities from the political and the economic. Yet, given the structure and funding of the project, we needed to do so. That led to constant reassessment of what was considered relevant. We had to make decisions as to whether we would include research on the diaspora (we decided not to do so), whether to include UN reports (we did), and how to access important scholarly grey literature produced by regional NGOs and research centers/institutes and the like (we developed a list of key NGO’s and research centers/institutes to contact one by one).

Once we completed the coding of Sociological Abstracts, we addressed duplicates.  Many articles fit easily in multiple thematics and therefore the same article often came up in the searches for several of the thematics. For example, an article could cover religion, class, media, sexuality. It would have been discovered in the searches for all those thematics. The combined database, however, had to eliminate the duplicates. Two GSRs were assigned the task of merging the data from the six thematics and deleting all the duplicates. We found it was less time-consuming do this all at once rather than trying to do it separately thematic by thematic. Ultimately, they found that uploading the databases to Covidence made deletion of duplicates easier, faster, and more efficient.

The hundreds of thousands of results from the searches boiled down to ~4,000 RIS files (Research Information System Citation file) when duplicates and irrelevant files were deleted. Coding the ~4,000 items on Zotero turned out to be a bear. Each of the six thematics were coded separately.  As we coded we discovered additional edits and corrections that needed to be made to the Code Book. It took over multiple months for five GSRs working at least 10 hours a week to finish the coding. Each RIS file took ~5 minutes to code. Even as we periodically checked coding, we continually discovered that references could have been coded many ways. We decided it was better to over code rather than under code – meaning that if a reference, based on the abstract, looked like it covered a topic, we coded it for that topic.  Coding updates led to constant updates in the Code Book.

We had considered archiving the database in a data repository. We consulted with UC Davis librarians on free services and for pay services. We considered Inter-University Consortium for Political and Social Research (ICPSR) at the University of Michigan, Ann Arbor.  Dataverse was recommended, as perhaps more suited to our needs.  ACSS has Dataverse, but mostly for quantitative data. ACSS thought they could adapt it to qualitative data as well. We turned to planning with ACSS and their web designers.

Parallel to the coding of Sociological Abstracts search results, a subcommittee of the team, including Suad Joseph, Meghan Klasic, and Zina Zarour, worked with the Arab Council for the Social Sciences and their subcontractor, WonderEight, to develop the webpage, specifically Youssef Kasab (WonderEight), Lynn Bitar (WonderEight), and Wassim Alam (ACSS). The initial plan was to house the database at multiple sites, including ACSS and the six partner universities. We learned that the cost of doing so would be prohibitive. Secondly, we realized that having the database in multiple sites would complicate updating and editing it. We decided that the database would be housed only at ACSS with links to all partner institutions.

The work to design the webpage took much of 2022, with support from ACSS, budgeted in the OSF grant. The partnership worked smoothly.  It was particularly helpful to learn that ACSS web designers use WordPress as their content management system (CMS). WordPress (which is used by 43% of websites, globally) organizes search functions by any word that a user chooses. If that word is in the database, Word Press will bring it up. Knowing that users would have two avenues for searches (using our topic words, or Word Press) eased the concern about the completeness of the coding in Zotero. WordPress themes and plugins offer flexibility.

Once the searches and coding of Sociological Abstracts was completed, we decided to search five more databases: JSTOR (EBSCO management); Project Muse; Anthropology Plus (EBSCO); Women’s Studies International (EBSCO); and BASE, Bielefeld Academic Search Engine.  We assigned three of the GSRs to do the searches with each of these databases, with each GSR now responsible for two of the six thematics.  They were to store the RIS files in Box folders specific to them and specific to the database and specific to the thematic. The other two GSRs worked on developing the database, setting up the tools for Zotero, moving the files from Box to Zotero, deleting duplicates in Zotero, once each thematic and database was completed.

Problems emerged. We had modeled our search and coding operation based on Sociological Abstracts. These additional databases had different management programs and, unlike Sociological Abstracts, a number did not provide abstracts. We had relied on the abstracts in Sociological Abstracts for the coding. We had to rely on titles alone, as it was far too time consuming to search for and read whole articles or books. This offered a critical lesson on how academic titles are often unrepresentative of the substantive topics of the research.

Jstor and BASE would not accept our Boolean Operators – our string of terms was too long for their database to handle. This required us to cut our Boolean operators, which compromised comparability. It would not allow us to select title and abstract for the references, but required us to select “All Fields.” They would not allow us to select English, French and Arabic – but only either All Languages or one specific language, further adding to search time.

Since our search protocols were developed for Sociological Abstracts (Proquest management) we decided to explore and search the databases under Proquest management and then adapt our search protocols for EBSCO and additional program management. We searched ERIC, Gender Watch, Index Islamicus, Language/Linguistics, and PsychInfo. We completed those searches in a couple of months as our protocols had been developed and worked well. Databases under EBSCO management included Anthropology Plus, Women’s Studies International, and Historical Abstracts. We explored JSTOR, but given its date limit and that it was mainly full articles, we decided against including it.

As we learned that some program management platforms claimed intellectual property over the abstracts, we decided not to include the abstracts from those platforms. We included abstracts from platforms that did not make such claims. This increased the importance of the coding and Code Book. Even with Word Press as the database manager, users would only be able to search titles and authors where we were not able to include abstracts. Our earlier decision to over code became prescient.

Coding was consuming more time than any of our activities. We developed a program for automatic coding based on our Boolean Operators, thanks especially to our GSR, Meghan Klasic. We used the automatic coding program (R) for ERIC, Gender Watch, Index Islamicus, Linguistics and Language Behavior Abstracts (LLBA), and PsychInfo.  We returned to Covidence for deleting duplicates from these five databases. We uploaded cleaned RIS files to Zotero.  We returned the RIS files Box to hold files to remove the irrelevant files.

While developing the automatic coding algorithms, we realized that we had many articles, relevant to our six thematics that were relevant to key words we had not searched. We decided to add those key words for the user to be able to retrieve these articles, but coded them in a different color in the Code Book. This took meticulous checking of the key words under each thematic and testing them against the range within the RIS files. At this point, the Code Book had been revised dozens of times.

We decided to include, for each RIS file, the link to the source database from which we gleaned the reference (of course many RIS files were in multiple databases, but we chose they key ones). Unfortunately, the link to the source database would work only if the user had a subscription or their institution had a subscription to the platform that housed the database (for example, Proquest, or EBSCO, BASE). We added a note to the introduction to the database to clarify the access question.

The databases on the Proquest platform responded well to our search protocols and the Boolean Operators. Although we had to adapt the protocols slightly for the EBSCO platform (Gender Watch International, Anthropology Plus, Historical Abstracts), our protocols and Boolean Operators worked relatively well on the EBSCO Platform.  The Bielefeld database (BASE) is one of the best for scholarship in European Languages, and lives on its own platform. As we launched the searches, we discovered that our search BASE would not accept our long strings of Boolean Operators. We explored work arounds for BASE.

The work with the Arab Council for the Social Sciences (ACSS) and their subcontractor, WonderEight Offshore Sal to design the website continued parallel to the database searches and coding.  This process took most of 2022. In September 2022, we invited a small group of faculty and graduate students to test the database, based on Sociological Abstracts, on the MPK page on the ACSS website. With their feedback, we made further edits. We developed a manual outlining how we developed the database. The manual, housed at ACSS, will offer the tools for updating and maintaining the database.

References

Joseph, Suad. 2022 “Categorical Boundaries: The Political Production of Kinship and Citizenship.” Citizenship Studies. DOI: 10.1080/13621025.2022.2103966