Required citation:

FAO. 2022. An indirect estimation approach for disaggregating SDG indicators using survey data – Case study based on SDG Indicator 2.1.2. Rome. https://doi.org/10.4060/cb8670en

The designations employed and the presentation of material in this information product do not imply the expression of any opinion whatsoever on the part of the Food and Agriculture Organization of the United Nations (FAO) concerning the legal or development status of any country, territory, city or area or of its authorities, or concerning the delimitation of its frontiers or boundaries. The mention of specific companies or products of manufacturers, whether or not these have been patented, does not imply that these have been endorsed or recommended by FAO in preference to others of a similar nature that are not mentioned.

The views expressed in this information product are those of the author(s) and do not necessarily reflect the views or policies of FAO.

ISBN 978-92-5-135785-9
E-ISBN 978-92-5-137273-9 (MOBI)
© FAO, 2022

Some rights reserved. This work is made available under the Creative Commons Attribution-NonCommercial-ShareAlike 3.0 IGO licence (CC BY-NC-SA 3.0 IGO; https://creativecommons.org/licenses/by-nc-sa/3.0/igo/legalcode).

Under the terms of this licence, this work may be copied, redistributed and adapted for non-commercial purposes, provided that the work is appropriately cited. In any use of this work, there should be no suggestion that FAO endorses any specific organization, products or services. The use of the FAO logo is not permitted. If the work is adapted, then it must be licensed under the same or equivalent Creative Commons licence. If a translation of this work is created, it must include the following disclaimer along with the required citation: “This translation was not created by the Food and Agriculture Organization of the United Nations (FAO). FAO is not responsible for the content or accuracy of this translation. The original [Language] edition shall be the authoritative edition.”

Disputes arising under the licence that cannot be settled amicably will be resolved by mediation and arbitration as described in Article 8 of the licence except as otherwise provided herein. The applicable mediation rules will be the mediation rules of the World Intellectual Property Organization http://www.wipo.int/amc/en/mediation/rules and any arbitration will be conducted in accordance with the Arbitration Rules of the United Nations Commission on International Trade Law (UNCITRAL).

Third-party materials. Users wishing to reuse material from this work that is attributed to a third party, such as tables, figures or images, are responsible for determining whether permission is needed for that reuse and for obtaining permission from the copyright holder. The risk of claims resulting from infringement of any third-party-owned component in the work rests solely with the user.

Sales, rights and licensing. FAO information products are available on the FAO website (www.fao.org/publications) and can be purchased through publications-sales@fao.org. Requests for commercial use should be submitted via: www.fao.org/contact-us/licence-request. Queries regarding rights and licensing should be submitted to: copyright@fao.org.

Contents

Acknowledgements

Abbreviations and acronyms

1. Introduction

2. Possible approaches to data disaggregation for SDG indicators based on survey data

2.1 Addressing data disaggregation at the analysis stage

3. The projection estimator

3.1 The basic approach: estimation of a population total

3.2 First extension: domain estimation

3.3 Second extension: estimation of a proportion or a ratio

4. Step-by-step implementation of the projection estimator with a case study based on SDG indicator 2.1.2

4.1 SDG Indicator 2.1.2

4.2 Presentation of datasets used for the case study

4.3 List of steps for the implementation of the case study

4.4 Recoding the variable of interest

4.5 Identifying and recoding potential auxiliary variables

4.6 Selecting the auxiliary variables to be included in the model

4.8 Estimating the projection parameters in the small sample

4.9 Computing the synthetic values in the large sample

5. Conclusions and way forward

Annexes

Annex A: Projection estimator on microdata from Guatemala

Annex B: Projection estimator on microdata from South Africa

References

Tables

1. Variables included in the GWP dataset (Malawi, 2016)

2. Cross-tabulation of the probability of being moderately or severely food insecure with the probability of being severely food insecure

3. Recoding of education categories of the Fourth Integrated Household Survey

4. Results of logistic regression for probability of moderate and severe food insecurity

5. Results of logistic regression for the probability of severe food insecurity

6. Projected versus direct estimates of the probability of being moderately or severely food insecure

7. Projected versus direct estimates of the probability of being severely food insecure

8. Cross-tabulation of the probability of being moderately or severely food insecure with the probability of being severely food insecure (Guatemala)

9. Recoding of ENCOVI’s education categories (Guatemala)

10. Results of logistic regression for the probability of being moderately or severely food insecure (Guatemala)

11. Results of logistic regression for being severely food insecure (Guatemala)

12. Projected versus direct estimates of the prevalence of moderate or severe food insecurity (Guatemala)

13. Projected versus direct estimates of the prevalence of severe food insecurity (Guatemala)

14. Cross-tabulation of the probability of being moderately or severely food insecure with the probability of being severely food insecure (South Africa)

15. Recoding of National income dynamics study education categories (South Africa)

16. Results of logistic regression for the probability og being moderately or severly food insecure (South Africa)

17. Results of logistic regression for the probability of severe food insecurity (South Africa)

18. Projected versus direct estimates of the prevalence of moderate or severe food insecurity (South Africa)

19. Projected versus direct estimates of the prevalence of severe food insecurity (South Africa)

Figures

1. The projection estimator to combine two independent surveys

2. Histograms of the probability of being 1) moderately or severely food insecure and 2) severely food insecure

3. Level of importance of the auxiliary variables for moderate or severe food insecurity

4. Level of importance of the auxiliary variables for severe food insecurity

5. Importance of different levels of auxiliary variables for moderate or severe food insecurity

6. Importance of different levels of auxiliary variables for severe food insecurity

7. Histogram of the probability of being 1) moderately or severely food insecure and 2) severely food insecure (Guatemala)

8. Importance of various levels of auxiliary variables for moderate or severe food insecurity (Guatemala)

9. Importance of various levels of auxiliary variables for severe food insecurity (Guatemala)

10. Histogram of the probability of being 1) moderately or severely food insecure and 2) severely food insecure (South Africa)

11. Importance of various levels of auxiliary variables for moderate or severe food insecurity (South Africa)

12. Importance of the levels of auxiliary variables for severe food insecurity

Boxes

1. Recoding the variables of interest

2. Recoding the variable sex in the two datasets (small and large sample)

3. Recoding the variable age in the two datasets (small and large sample)

4. Recoding the variable education in the two datasets (small and large sample)

5. Recoding the variable employment in the two datasets (small and large sample)

6. Recoding the variable rural in the two datasets (small and large sample)

7. Recoding the variable income quintile in the two datasets (small and large sample)

8. Recoding the variable marital in the two datasets (small and large sample)

9. Implementing Boruta with R for the probability of being moderately or severely food insecure

10. Implementing Boruta with R for the probability of being severely food insecure

11. Implementing weighted logistic regression with R

12. Projecting the synthetic values in the large sample (Fourth Integrated Household Survey) with R

13. Indirect estimation of the probability of being moderately or severely food insecure and its variance with R

14. Indirect estimation of the probability of being moderately or severely food insecure and its variance with R

Acknowledgements

This technical report, which further expands the methods and case studies presented in the FAO Guidelines on data disaggregation for Sustainable Development Goal (SDG) Indicators using survey data (FAO, 2021), was prepared by the Office of Chief Statistician (OCS) of the Food and Agriculture Organization of the United Nations (FAO), under the general direction and encouragement of FAO Chief Statistician, Pietro Gennari.

Piero Demetrio Falorsi supervised the development of the entire publication, while the individual sections were drafted by the following authors:

• Section 1: Clara Aida Khalil and Piero Falorsi;

• Section 2: Piero Falorsi and Clara Aida Khalil;

• Section 3: Piero Falorsi and Clara Aida Khalil;

• Section 4: Clara Aida Khalil, Stefano Di Candia and Ayça Dönmez;

• Section 5: Piero Falorsi and Clara Aida Khalil.

The authors are particularly grateful to Carlo Cafiero, Sara Viviani, for their precious technical inputs, and suggestions for current and future improvement of the study.

Abbreviations and acronyms

FAO Food and Agriculture Organization of the United Nations

FIES Food Insecurity Experience Scale

FIES- SM Food Insecurity Experience Scale Survey Module

GWP Gallup World Poll

HT Horvitz-Thompson

IAEG-SDG Indicators Inter-Agency and Expert Group on Sustainable Development Goals indicators

IHS Integrated Household Survey

IHS4 Fourth Integrated Household Survey

IRT item response theory

NSO National Statistical Office

OCS Office of Chief Statistician

prob.ms probability of moderate and severe food insecurity

prob.s probability of severe food insecurity

SDG Sustainable Development Goals

UNSC United Nations Statistical Commission

WM Working model

1. Introduction

As the custodian United Nations (UN) agency of 21 Sustainable Development Goal (SDG) indicators, and a member of the Inter-Agency and Expert Group on Sustainable Development Goals indicators (IAEG-SDGs) and the Working Group on data disaggregation, the Food and Agriculture Organization of the United Nations (FAO) has been working to support countries in reporting SDG indicators at the required disaggregation level. To this end, FAO Office of Chief Statistician (OCS) has developed Guidelines on data disaggregation for SDG Indicators using survey data (FAO, 2021; from here on referred to as “the Guidelines”), which offer methodological and practical guidance for the production of direct and indirect estimates of SDG indicators having surveys as their main or preferred data source. Starting from this work, the FAO is continuing working on data disaggregation and indirect estimation approaches, by developing practical case studies on indicators under its custodianship.

As in the Guidelines, this technical report presents a case study based on the so-called “projection estimator”, allowing the integration of two independent surveys for the production of synthetic disaggregated estimates. In particular, the publication presents a practical exercise focused on the production of disaggregated estimates for SDG Indicator 2.1.2, on the Prevalence of Moderate or Severe Food Insecurity in the population based on the Food Insecurity Experience Scale (FIES). This application – based on survey microdata from Malawi – expands and enriches the brief practical exercise presented in the Guidelines by:

• Providing a step-by-step guide for the replication of the exercise and its implementation on different datasets and contexts.

• Illustrating the basic R routines developed for each of the steps, in order to promote and facilitate the adoption of the open source software in countries’ national statistical offices (NSOs).

• Presenting the functional form of the projection estimator necessary to address all the SDG indicators under FAO custodianship the computation of which should be based on survey microdata.

• Providing the necessary theoretical and practical tools to assess the precision and accuracy of the produced indirect disaggregated estimates;

• Replicating the case study on survey datasets from two additional countries, namely Guatemala and South Africa, the results of which confirm the robustness of the approach and are reported in separate annexes of this publication.

Given the practical nature of this technical report, the target audience includes statistical practitioners in NSOs and international organizations wanting to adopt the projection estimator for data disaggregation of SDG indicators by means of integrating survey data with additional data sources such as other surveys, censuses, administrative records, and/or geospatial information.

The report is structured as follows. Section 2 provides an overview of the main challenges to achieve data disaggregation for SDG indicators having survey data as their main data source. In addition, the section presents possible strategies to address these limitations at the analysis stage of the statistical production process. Section 3 illustrates in detail the main characteristics of the indirect estimation approach based on the projection estimator, highlighting its relevance in the context of the SDG monitoring Framework.

Section 4 provides a step-by-step guide on how to implement the projection estimator, using an example based on indicator 2.1.2 and microdata from Malawi. Finally, Section 5 presents the main conclusions from the study and outlines possible extensions and way forward. In addition to the case study presented in Section 4, Annex A and B present results for the same indirect estimation method implemented on microdata from Guatemala and South Africa.

2. Possible approaches to data disaggregation for SDG indicators based on survey data

In a sample-survey context, the estimator of a parameter of interest for a given subpopulation is said to produce a direct estimate when the estimation process is based only on sample information from the subpopulation itself.

Unfortunately, for most surveys, the sample size is not large enough to guarantee reliable direct estimates for all subpopulations. In addition, the sample of most surveys does not cover all possible sub-domains of the population, and there is the possibility of having disaggregation domains without any sample observation. A “small area” or “small domain” is any subpopulation for which a direct estimator with the required precision cannot be obtained with a given data source.[1]

These issues can potentially be addressed at different stages of the statistical production process. They can be tackled at the design stage, by adopting sampling strategies guaranteeing an observed set of sampling units for every subpopulation for which disaggregated data must be produced. With traditional sampling techniques, this goal implies increased survey costs and complexity and can become quickly unfeasible when dealing with multiple disaggregation domains. On the other hand, problems of this nature can be addressed at the analysis stage, by adopting approaches of indirect estimation that cope with the little information available for small areas by borrowing strength from additional data sources or domains.

The Guidelines on data disaggregation for SDG indicators using survey data (FAO, 2021) provide a detailed review of methods to deal with data disaggregation at the sampling design, and also discusses methods to be adopted at the analysis stage. Referring to that publication for a complete overview, the main categories of approaches to address data disaggregation during the analysis phase is provided below.

2.1 Addressing data disaggregation at the analysis stage

Data disaggregation can be addressed adopting indirect estimation approaches coping with the little information available for so-called small areas, by borrowing strength from additional domains. In particular, the integrated use of different data sources offers a powerful approach for achieving the desired level of disaggregation by preserving estimates accuracy.

Typical data sources that could be integrated with data from a particular household and/or agriculture surveys are:

• other surveys;

• censuses;

• administrative registers;

• geospatial information and big data.

Indirect estimation approaches range from model-based methods to model-assisted approaches.

The model-based approach (such as that adopted in small area estimation techniques) assumes that the values of a variable of interest observed on the units of a population are the realization of a random variable. The model (often denoted with the term superpopulation model) defines a class of distributions to which this random variable is supposed to belong. In this context, the sample is interpreted as the result of a double random experiment:

1) the observed realization of the model generates the population from which the sample is drawn; and,

2) the sample units are observed in accordance with specific random selection rules incorporated in the sampling designs (Royall, 1976; Valliant et al., 2000; Chambers and Clark, 2015; and Tillé, 2019).

Taking into account the fact that the sample is an ancillary statistic[2], Royall (1976) proposed to develop the inference conditionally on it. Indeed, once the sample is selected, the observed units are no longer random.

In these contexts, it is important that the model express a known and previously tested relationship. If the model adequately describes the population, inference can be conducted with respect to the model and conditional to the sample selection. In other words, when the model is correct, a model-based approach results in the optimal estimator. However, a model is always an approximate representation of reality. For instance, the model may fail in its objective to reproduce reality when the survey does not capture some relevant auxiliary variables for the phenomenon at hand. Hansen et al. (1983) argue that, when the model is not correctly specified, the bias may be so important to result in confidence intervals that do not include the true value of the parameter to be estimated.

The debate on the validity of model-based approaches versus those based on the properties of sampling designs is wide and interesting, even if its arguments are more philosophical than mathematical. From a statistical point of view, both theories are valid. The controversy relates to the idea that, with a model, we provide a formalization of reality that may not be the correct one. The principle of impartiality – which is one of the fundamental principles of official statistics – is a strong argument against the adoption of inferential approaches entirely based on model assumptions.

As a response to this debate, a hybrid approach – the so-called model-assisted approach – was developed, which allows producing valid inference under model assumptions and is robust to wrong specifications of the model. In this context, the model only allows exploiting auxiliary information available at the time of estimation to increase the accuracy of final estimates and deal with traditional issues such as non-response. At the same time, the estimation process is based on the inferential properties of the survey’s sampling design. In model-assisted approaches, the model is often denoted as Working model (WM), thus meaning that it does not have the ambition to explain the phenomenon, but only to offer a working tool that can help improving the quality of the estimator. For more details on model-assisted approaches, the authors of this technical report refer to Särndal, Swensson and Wretman (1992), where it is demonstrated that an estimator developed under a model-assisted approach is approximatively unbiased under the assumption of repeated sampling, irrespective of the shape of the finite population scatter. From this follows that the estimator is unbiased irrespective of whether the assumptions of the model are true or false. On the other hand, the validity of the model is a crucial factor to achieve a small variance.

3. The projection estimator

Among the various approaches available to produce indirect disaggregated estimates for SDG indicators, this technical report presents and applies a model-assisted approach based on the so-called projection estimator. This method was first presented in its simpler formulation in the seminal paper “Combining data from two independent surveys: a model-assisted approach” from Kim and Rao (2012).

The paper starts from the following scenario: Let us consider two independent surveys, where the first survey is characterized by a large sample A_1, and collects only variables of general use (auxiliary variables); while the second survey has a smaller sample A_2, but collects information on a target variable y along with the same set of auxiliary variables available in A_1. The two samples A_1 and A_2 may be selected from possibly different frames exploiting distinct sampling designs. (Kim and Rao, 2012). A relevant assumption is that the auxiliary variables collected with the two surveys share similar structure and definitions.

Kim and Rao (2012) suggest a model-assisted projection method of estimation based on a WM that results in asymptotically unbiased projection estimators. With this approach, a two-steps process (Figure 1) generates synthetic or proxy values of a variable of interest:

• First, the WM linking the variable of interest to the common auxiliary variables is fitted on data from A_2, leading to the estimation of a set of model parameters.

• Then, the values of the variable of interest are predicted in A_1 by applying the estimated parameters to the auxiliary variables observed in the large sample.

In the following paragraphs we introduce the basic theory and notation used in Kim and Rao (2012), presenting also some possible extensions considered relevant in the context of the SDG monitoring framework.

3.1 The basic approach: estimation of a population total

Let Y = yi be a target population total, where yi is the value of the variable of interest y on unit i and N the overall population size. A Working model is introduced according to which yi = m(xi; β) + ui, where ui is a random residual and m(xi; β) is a known function applied on the column vector of auxiliary variables xi, with β being the column vector of the model parameters. The two column vectors xi and β must have the same number of elements.

The model expectation of ui is equal to 0. i.e. EM (ui) = 0.

When the WM is a regression model, m(xi; β) = β, with being the transpose of xi.,

Let be the estimator of β obtained from the second survey, using data {(yi; xi): i ∈ A_2} and let = m(xi; ) be the predicted value of yi, with EM() = β. Let EP denote the expectation under repeated sampling and let wi1 be the sampling weights of the sample A_1 allowing to compute sampling-unbiased estimates.

If yi would be available in A_1, = yi would be a sample unbiased estimator of Y, where the sampling unbiasedness implies that, under the repeated sampling, the expected value of is equal to the unknown total Y: EP( − Y) = 0.

However, the estimator cannot be implemented from sample A_1, unlike the following estimator of Y, which is based on the synthetic values = m(xi; ) projected in the second sample:

The estimator is called projection estimator (or synthetic estimator), as = m(xi; ) can be viewed as a projection of yi using the auxiliary variables xi.

Bias and variance

The estimator is unbiased with respect to both the model and the sampling design, as follows:

The asymptotic sample bias of is

with β_0 denoting the estimate of β when observing the entire population, i.e. the estimation that we would get using census data.

The asymptotic sample bias from the second survey can be estimated as

where wi2 are the sampling weights of A_2, which allow for computing sample-unbiased estimates for the second survey.

Thus, is not sample-unbiased, except when

Therefore, to guarantee sample-unbiasedness, estimates of should be obtained by respecting the condition established in Formula 3.2. For generalized linear models (such as heteroscedastic linear regression models or logistic models) to satisfy condition 3.2, it is assumed that the first element of xi is equal to unity, which means that the model has an intercept.

Kim and Rao (2012) demonstrate that the sample variance of is given by

where the first term on the right-hand side is the variance due to sampling, in survey S_1, of the population predictions (with the β_0 value), and the second term is the variance due to sampling, in survey S_2, of the population residuals (for the predictions with the β_0 value). The latter term tends to be small if the residuals are small, i.e. if model m is sufficiently predictive.

We can derive a plug-in asymptotically unbiased estimator of Var () by substituting the population value β_0 with the estimate , as reported below:

where ar(·) denotes the sampling estimate of Var(·).

3.2 First extension: domain estimation

Let d denote a particular domain for which disaggregated data must be produced (e.g. sex of individuals, the indigenous status, or a particular geographic location).

Let

be the total of the target variable for the d-th domain, where γdi is the domain membership variable such that:

The projection estimator of the total Yd is given by:

The condition for sample-unbiasedness becomes

To satisfy Condition 3.6, vector xi must include the γdi values, which means that the model has a domain intercept. This can be fulfilled only if, in sample A_2, domain d has a sufficient sample size.

However, in general, the condition established under condition 3.6 cannot be ensured in the sampling design phase for very small disaggregation domains. Therefore, it is preferable to focus on the model conditions that provide negligible bias. From Kim and Rao (2012), it can be derived that the relative bias EP( − Yd)/Yd can be expressed as

where Cov[γdi, (yi − m(xi; β))] is the population covariance between the domain membership indicators, γdi, the model residuals yi − m(xi; β), is the population mean of the domain membership indicators, and is the population mean of the product variable γdi yi.

Therefore, to make sure that the relative bias is close to 0. the model should be specified to ensure that the model residuals depend slightly on the domain membership variables:

This will be the case if the WM is correctly specified.

From the relationship in Formula 3.7, it can also be seen that in large domains, for which is large, the relative bias becomes negligible.

Finally, the variance can be obtained easily from Expression 3.3 as

Where the sampling estimate is

Highlight: It is possible to produce cross-tabulations of the variable of interest y also for disaggregation domains not included in the data collection instrument used to get A_2 (sample providing information on y). For example, let’s suppose to be interested in estimating a parameter related to y, disaggregated by indigenous status. Let us also assume that the information on the indigenous status of respondents is not available in A_2, but only in A_1. By projecting the values of y on A_1, it is possible to use the auxiliary information on the indigenous status to estimate the parameter of interest considering this disaggregation dimension.

3.3 Second extension: estimation of a proportion or a ratio

In many cases, SDG indicators based on survey data present the following functional form:

where yd is defined as in Section 3.2 (formula 3.4) and

zi being the value of the variable z on unit i, where the variable z is observed in the survey A_1.

In all these cases, the projection estimator can also be expressed in the form of the ratio:

where is defined in Section 3.2 (formula 3.5) and

is the direct estimate of the total Xd from the survey A_1.

When zi = 1, expression (3.11) provides the projection estimator of a proportion

where

is the direct estimator of the population size in domain d

In order to study the asymptotic properties of estimator (3.11), we consider its linear approximation, given by the first order terms of Taylor’s series approximation:

where oi is a rest of minor order, and

Omitting the term oi, we have that - for large values of N and n - the estimator is approximately design unbiased, and the variance is

To have a rough evaluation of the variance of as a function of the variances of the numerator and denominator, we can adopt the approximation ≅ Rd.

From this it follows that:

In Woodruff (1971) it is demonstrated that this expression may be approximated with the variance of the total of a transformed variable z:

where tdi is the Woodruff transformation:

A plug-in estimate of tdi from the survey data is

The plug-in sampling estimate of Variance 3.15 is

Highlight: Various SDG Indicators under FAO custodianship can be treated using the ratio extension of the projected estimator. These are:

SDG Indicator 2.1.1: Prevalence of Undernourishment;

SDG Indicator 2.1.1: Prevalence of moderate or severe food insecurity in the population based on the FIES;

SDG Indicator 2.3.1: Volume of production per labour unit by classes of farming / pastoral / forestry enterprise size;

SDG Indicator 2.3.2: Average income of small-scale food producers, by sex and indigenous status;

SDG Indicator 5.a.1.a (Percentage of people with ownership or secure rights over agricultural land (out of total agricultural population), by sex) and 5.a.1.b. (share of women among owners or rights-bearers of agricultural land, by type of tenure)

4. Step-by-step implementation of the projection estimator with a case study based on SDG indicator 2.1.2

This section illustrates the steps and the software to integrate two independent surveys to produce disaggregated estimates, by means of a practical example relying on the projection estimator discussed in Section 3. This model-assisted indirect estimation approach is applied to SDG Indicator 2.1.2 on the prevalence of moderate or severe food insecurity in the population based on the FIES. Even though all the steps for implementing the projection estimator are presented with reference to a particular indicator, this approach has a much wider applicability and could be adapted to other SDG indicators based on survey and/or census data.

4.1 SDG Indicator 2.1.2

SDG Indicator 2.1.2 provides internationally comparable estimates of the percentage of individuals in the population who have experienced food insecurity at moderate or severe levels during a pre-defined reference period. The measurement of the level of severity of food insecurity is based on the FIES, which is an experience-based metric of food insecurity severity (FAO, 2022).

The scale relies on people’s direct responses to questions about their experiences facing constrained access to food. The Food Insecurity Experience Scale Survey Module (FIES-SM) is composed of eight questions with simple dichotomous responses (yes/no). Respondents are asked whether, at any time during a certain reference period, they have experienced any of the subjective dimensions of food insecurity captured by the scale due to limited availability of money or other resources. Responses to these eight questions are analyzed using the item response theory (IRT) to obtain cross-country comparable measures of the severity of food insecurity of individuals, treated as a “latent” – or unobservable - trait.

Unfortunately, too few countries have currently included the FIES module in their nationally representative surveys. To fill this data gap, in 2015, the FAO began collecting the FIES data by leveraging on the Gallup World Poll (GWP), a branch of Gallup, Inc. that surveys nationally representative samples of the adult population annually in nearly 150 countries, covering 90 percent of the world’s population. This has enabled FAO to collect information from individual respondents at a relatively low cost and to compute country-level estimates of the prevalence of food insecurity at different levels of severity that are valid, reliable and comparable across countries.

For what concerns data disaggregation, this is particularly relevant to properly monitor the prevalence of moderate or severe food insecurity in the population. To ensure regular access to nutritious and sufficient food, and thus reduction of food insecurity, detailed and disaggregated information by age, gender, income level, and geographic location is necessary to identify priority efforts and interventions. Besides dimensions recommended by the IAEG-SDGs[3], additional disaggregation dimensions may be relevant in specific country contexts.

In order to produce direct disaggregated estimates of Indicator 2.1.2 by the above reported and other disaggregation dimensions, two conditions need to be satisfied:

• The stratification or disaggregation variables - such as sex, age, income, education, and geographic location – have to be available in the survey dataset;

• The sampling size needs to be large enough to produce accurate estimates for each disaggregation dimension.

The first condition is satisfied also when data are collected through the GWP for standard disaggregation dimensions such as sex, age, income quintile, geographic location, and education. However, given the limited number of auxiliary information available in the dataset, other dimensions that may prove relevant at the national level would not be possible. For example, information on the indigenous and/or migratory status of individuals is not available in the GWP dataset. The second aspect is often more problematic. Indeed, GWP samples are large enough to guarantee representative and accurate estimates at the national level, but often too small to ensure accuracy at more detailed disaggregation level.

These two types of issues can be addressed with the indirect estimation approach discussed in Section 3 and tested in this technical report.

4.2 Presentation of datasets used for the case study

The projection estimator presented in Section 3 has been tested on microdata from two surveys implemented in Malawi in 2016.

4.2.1 Small sample: FIES individual module collected through the Gallup World Poll (GWP)

As mentioned in Section 4.1, in 2014, the FAO started collaborating with the Gallup Inc. to implement the FIES module in over 150 countries. The FIES-SM collects information on the experience of people (individuals over the age of 15) with food insecurity, through annual nationally representative samples (of a size of approximately 1 000 individuals). In the case of Malawi, the FIES module was translated in the two local languages (Chichewa and Chitumbuka) to make sure that the intended meaning of each question was rightly expressed. The Gallup dataset for 2016 includes a sample of 1 000 individuals divided in 125 primary sampling units.

Variables in the dataset are described in Table 1 below. Variables reported in yellow cells contains dichotomous answers (Yes/No) provided by individual respondents to questions in the FIES-SM. Variables in green cells are instead those used to build auxiliary variables for fitting the projection model. Finally, variables in pink cells have been used to construct the dependent variables of the two fitted projection models (see Section 4.4). It should be noted that, compared to the exercise presented in the Guidelines (FAO, 2021), this case study relied on an extended version of the GWP dataset including additional auxiliary variables, namely the marital status of respondents and a dummy indicating whether they were in possession of a mobile phone.

The categories of above mentioned auxiliary variables are detailed in Section 4.3, where the adopted recoding procedure is also illustrated.

4.2.2 Big sample: The Fourth Integrated Household Survey (IHS4) of Malawi (2016–2017)

The HIS is implemented by the NSO of Malawi every three years to monitor and evaluate the changing conditions of Malawian households. This survey is an important source of information on the country’s socio-economic indicators, which are key to the evidence-based policy formulation process and monitoring progress towards achieving the SDGs.

The IHS4 is the fourth full survey conducted under the umbrella of the World Bank’s Living Standard Measurement Study – Integrated Surveys on Agriculture (LSMS-ISA), and was fielded from April 2016 to April 2017. The IHS4 collected information from a sample of 12 480 households statistically designed to be representative at national, district, and urban/rural levels.[4] The cross-sectional survey used four questionnaires: 1) the household questionnaire, 2) the agricultural questionnaire, 3) the fishery questionnaire, and 4) the community questionnaire. Full detail on the scope and structure of these four questionnaires is provided in the survey report (NSO, Malawi, 2017) and in the WB microdata catalogue (World Bank Microdata Library). Module T of the household questionnaire was designed to perform a subjective assessment of household’s wellbeing, including also the FIES module at household level.

4.3 List of steps for the implementation of the case study

Exploiting microdata collected with surveys described in Section 4.2, the projection estimator has been used to produce disaggregated estimates of SDG Indicator 2.1.2, following the steps listed below:

1. Recoding the variable of interest. As discussed in Section 3, the synthetic values are predicted through a known function m(xi; ) of the estimator , which is obtained from the small sample (the GWP microdata in the case of Malawi). The selection of the functional form for m relies heavily on the type of variable y considered (e.g. scale, nominal, dichotomous). Section 4.4 shows how the variable of interest of this study was recoded to then apply a multinomial logistic regression.

2. Identifying and recoding auxiliary variables. The implementation of the projection estimator proposed by Kim and Rao (2012) requires the availability of the same set of auxiliary variables in the two surveys to be integrated. In order to improve the efficiency of the projection, these variables need also to share common structure and definitions. Based on this prerequisite, Section 4.5 illustrates variables that have been included as potential auxiliary variables for the implementation of the projection estimators, providing the R basic commands for recoding and harmonizing them in the two datasets.

3. Selection of variables to be included in the model. Among the various statistical approaches available to select auxiliary variables to be included in a regression model, Section 4.6 of this technical report discusses and applies the Boruta feature selection method from Kursa and Rudnicki (2010). In addition, the section illustrates the Boruta R package (Boruta) for the implementation of such method.

4. Definition of the function 𝒎() and estimation of projection parameters. In Section 4.8, a weighted multinomial logistic regression is used to estimate the projection parameters needed to compute synthetic values of the variable of interest. The implemented regression model includes both the auxiliary variables selected through Boruta and those representing the disaggregation dimensions of interest (age, sex, income and geographic location). This is done to meet the condition ensuring sample unbiasedness of the projection estimator for a specific projection domain.

5. Computation of synthetic values. Using the estimated projection parameter, Section 4.9 illustrates the computation of synthetic values of the variable of interest in the large dataset. This in turn, will allow producing disaggregated estimates of SDG Indicator 2.1.2 by all the considered disaggregation dimensions.

6. Assessment of estimates accuracy. Section 4.10 illustrates the R packages available to estimate the variance of the projected disaggregated estimates. The variance can be used as a measure of estimates accuracy.

4.4 Recoding the variable of interest

This study considers the following two variables of interest (dependent variables):

• Prob_Mod_Sev = The probability of being moderately or severely food insecure;

• Prob_Sev = The probability of being severely food insecure.

While the first variable is the base for computing the prevalence of moderate or severe food insecurity in the population (SDG Indicator 2.1.2), the second is another population parameter that can be estimated with the FIES module at a more severe level. Being two probabilities, these two variables could ideally take any value in the interval [0.1]. However, by construction, their distribution were concentrated around few values (Figure 2), i.e. the nine possible raw scores.

Hence, before implementing the projection estimator, it was decided to recode the dependent variables into categorical ones. Initially, the dependent variables were grouped into three categories to apply an ordinal regression model. However, using ordinal regression models brought more complexity into the overall estimation process, without (in this specific case) a significant improvement of final estimates. In addition, the results obtained with this type of regression model are not of easy interpretation.

As the objective here is to develop a flexible method of disaggregation that is also easy to implement, it was decided to use logistic regression with binary dependent variables. Hence, the two probabilities have been recoded into binary categorical variables, taking value 1 for probabilities higher than or equal to 0.5, and 0 otherwise. It must be noted that the so built individual variables approximately lead to the prevalence of food insecurity indicators at the two levels of severity when aggregated up for the full sample; however, they do not replicate exactly the value of the indicators as they are based on a transformation of the original respondent-level food insecurity variables. Box 1 shows the R syntax used to recode the two probabilities in the GWP dataset. Despite presenting basic R commands, this and the following boxes provided in this report, are intended to facilitate the reproducibility of results, and allow implementing the projection estimators on different datasets with a limited amount of modifications.

Out of the 1 000 respondents, five of them had a missing value for both probabilities, and were removed from the dataset.

4.5 Identifying and recoding potential auxiliary variables

As seen in Section 3, the implementation of the indirect estimation approach proposed by Kim and Rao (2012) requires having the same set of auxiliary variables in the two samples. In the small sample, this group of variables is used to fit a regression model for the estimation of a set of projection parameters. In turn, the auxiliary variables – along with the estimated projection parameters – allow predicting the synthetic values of the variable of interest in the big sample.

Concerning the two datasets used for this case study, the IHS4 provided access to a vast range of information collected with its four questionnaires. In contrast, the GWP only collects the FIES module at individual level along with a limited number of demographic, social and economic variables. Auxiliary variables that could be retrieved from both samples were: 1) sex; 2) age; 3) education; 4) employment status; 5) income quintile; 6) number of adults in the household; 7) number of children in the household; 8) marital; and 9) cellphone. Hence, the principal element in the selection of auxiliary variables for this case study was data availability.

One of the conditions to be satisfied by auxiliary variables before implementing the projection approach, is for these to share similar definitions and structure in the two samples. Hence, before implementing the indirect estimation approach, all the selected auxiliary variables have been recoded and harmonized across the two surveys. Boxes presented in this section provide examples of R syntax that could be used to perform variables recoding operations.

Recoding the sex of respondent

In both datasets, the variable sex has been recoded into the categorical dummy variable female taking value 1 for female respondents and 0 for male respondents.

Recoding the age of respondent

The GWP datasets contains individual answers to questions in the FIES module only for individuals who are 15 years old or older. Hence, the first step consisted in removing the observations corresponding to people below 15 years of age from the IHS4 dataset. Subsequently, the variable age has been recoded in both datasets according to the following age classes:

• agecat_1: 15-24 (youth)

• agecat_2: 25-49

• agecat_3: 50-64

• agecat_4: 65 and above.

Recoding education information

Information on education were collected with a very different level of detail in the two surveys. More precisely, while the IHS4 provided very granular information on household members’ education level, the GWP only distinguished between people with:

• educat_1: completed elementary education or less (up to 8 years of basic education);

• educat_2: completed secondary/three-year tertiary education and some education beyond secondary (9-15 years of education);

• educat_3: completed four years of education beyond high school and/or with a four-year college degree.

In addition, nine of the 1 000 respondents included in the GWP either refused to provide information on education or reported to not know that information. Given the limited number of cases and to simplify the recoding process, these two answer options have been included in educat_1 (which was the group with the highest frequency). For what concerns the education variable in the IHS4, Table 3 shows how initial categories have been recoded into educat_1, educat_2, and educat_3.

Recoding employment information

The variable Employment in the GWP dataset was originally coded as follows:

• employed full-time for an employer;

• full-time self-employed;

• employed part-time, wants full-time;

• employed part-time, does not want full-time;

• unemployed;

• out of workforce.

Variables on employment based on IHS4 microdata were extracted from the FAO Rural Livelihoods Information System (RuLIS) database. In this database, the variable “tot_employment” is coded as follows:

• 0 for not employed (inactive or unemployed);

• 1 for employed.

Hence, in order to harmonize information available in the two samples, the more detailed variable available in the GWP dataset has been recoded in the following dummy:

• empcat = 1: employed full-time for an employer; employed full-time for self; employed part-time, wants full-time; employed part-time, does not want full-time.

• empcat = 0: unemployed, out of workforce.

Recoding the geographic location

In both datasets, the dummy rural has been created, taking value 1 for individuals from towns and rural areas and 0 for individuals from urban areas and suburbs.

Recoding the variable household size

The variable household size has been created in different ways in the two datasets. For what concerns the GWP dataset, the total number of individuals in the household has been computed by summing the number of adults with the number of children in the household. On the other hand, for the IHS4, the household size has been extracted from the RuLIS by counting the number of individuals reported in the household roster by household.

Recoding the variable income quintile

The GWP dataset already provides a variable indicating the income quintile to which each individual belongs. On the contrary, income quintiles needed to be computed in the IHS4 dataset. As income aggregates are already produced in the context of the RuLIS, for this exercise we referred to the variable “tot_income”. Income at the individual level was estimated by dividing it by the number of adults in the household. As a result, RuLIS estimates for IHS4 are converted into categorical variables following the GWP definitions:

• inccat_1: Poorest 20%;

• inccat_2: 21% - 40%: Second 20%

• inccat_3: 41% - 60%: Middle 20%;

• inccat_4: 61% - 80%: Fourth 20% ;

• inccat_5: Richest 20%.

Recoding information on the marital status

Both the GWP and the IHS4 collected information on the marital status of household members. For what concerns the GWP, the variable Marital was initially coded as follows:

• 1 - Single/Never been married;

• 2 - Married;

• 3 - Separated;

• 4 - Divorced;

• 5 - Widowed;

• 6 - Domestic partner.

Similarly, the variable marital included in the IHS4 dataset had the following categories:

• 1 - Monogamous married;

• 2- Polygamous married;

• 3- Separated;

• 4 - Divorced;

• 5 - Widow or widower;

• 6 - Never married.

In order to harmonize the information in the two datasets, variables categories have been recoded in three classes, namely 1) never married; 2) married; 3) other. The recoding R syntax is illustrated in Box 8. It should be noted that, despite a more granular categorization of the variable marital was possible, in both cases, the categories 3) separated and 4) divorced were reported by very few respondents. For this reason, the single category other, including widowed, divorced, and separated individuals was created.

4.6 Selecting the auxiliary variables to be included in the model

One of the fundamental steps to ensure the quality of results obtained with the projection estimator is the identification of suitable auxiliary variables xi. In similar contexts, especially when big surveys collecting a multitude of information are considered, the use of variable selection methods can be helpful, and issues such as multicollinearity should be carefully assessed.

As mentioned in the Guidelines (FAO, 2021), the literature on variable selection approaches is very ample, with authors such Ryan (2008) and Harrel (2015) providing comprehensive summaries of the common methods used in this field.

In this study, despite the availability of a relatively small number of auxiliary variables common to the two datasets, we illustrate the use of the Boruta feature selection method, proposed in Kursa and Rudnicki (2010). For more details on the theory behind this selected approach, the authors refer to the Guidelines (FAO, 2021) besides, of course, the original paper presenting the approach (Kursa and Rudnicki, 2010).

For this application, all the auxiliary variables available in the GWP dataset were plugged into the Boruta algorithm to assess their relevance. The algorithm was implemented separately for the probability of being moderately or severely food insecure (Figure 3) and the probability of being severely food insecure (Figure 4).

In Figures 3 and 4, the boxplots of different colors represent various Boruta outputs: the red, yellow and green boxplots represent the scores of the rejected (unimportant), tentative and confirmed (important) variables respectively, while the color blue was assigned to shadow features. Tentative variables are those for which Boruta could not indicate a clear decision concerning their relevance, as their importance level was not significantly different from their best shadow features.

From Figure 3 emerges that the most important auxiliary variables to explain the probability of being moderately or severely food insecure are the income quintile, the geographic (urban/rural) location, the education level, the ownership of a mobile phone, and the sex of respondents listed in order of importance. On the other hand, information on the employment status, the age, and the household size are identified as unimportant to explain the variability of the variable of interest. In this case, Boruta did not identify any tentative feature.

Similarly, the most important auxiliary variables for the probability of severe food insecurity identified by Boruta are the income quintile, the ownership of a mobile phone, the geographic location, the education level and the size of the household. The variables agecat and empcat are indicated as tentative attributes, while the variable female is classified as not relevant (Figure 4).

In order to perform a more accurate analysis and identify levels of categorical variables with greater influence, the implementation of the Boruta algorithm was repeated on the various levels of available categorical auxiliary variables (see Section 4.5 for the definition of each level). To do so, a series of dummy variables, one for each category of all auxiliary variables, were created in both datasets.

Figures 5 and 6 below present the results of applying Boruta on these dummies for the two variables of interest. Concerning R, the syntax to be used is identical, except for the creation of dummy variables.

Figure 5 allows assessing which are the important levels of auxiliary variables confirmed by Boruta. Levels identified as important to explain the variability of the probability of being moderately or severely food insecure are:

• the rural dummy;

• educat_2 and educat_1 dummies; and

• the inccat_5 dummy.

Similarly, the levels of auxiliary variables identified as important by Boruta are:

• the inccat_5, inccat_1 and inccat_2 dummies;

• the educat_1, educat_2, and educat_3 dummies;

• the rural dummy; and

• the cellphone dummy.

In this case, Boruta identified as tentative the dummy variables agecat_1, marital_3, sizeHH, and empcat.

As illustrated in Section 4.8, all the levels of auxiliary variables identified as tentative or important by Boruta have been used to fit a logistic regression on the two variables of interest. In addition, all the relevant dimensions for data disaggregation (sex, age class, income, rural/urban location) have also been included in the regression model, in order to increase the sample unbiasedness of the projection domain estimator (see Section 3 for theoretical justification).

4.8 Estimating the projection parameters in the small sample

Two weighted multivariate logistic regressions have been implemented in the small sample to estimate the projection parameters to be used to predict the values of the two variables of interest in the large survey.

Let us indicate with the probability of being moderately or severely food insecure for the i − th individual in the small sample, and with the probability of being severely food insecure for the same individual. These probabilities were estimated using GWP data collected with the FIES individual module.

As seen in Section 4.4, and were concentrated around few discrete values in the [0.1] interval. For this reason, they were recoded into two dummy variables yms,i, and ys,i, where:

• yms,i = 1 if ≥ 0.5, and yms,i = 0 otherwise;

• ys,i = 1 if ≥ 0.5, and ys,i = 0 otherwise.

Then, the yl,i values (with l = ms or l = s) were modeled with a multivariate logistic function of the set of discrete categorical auxiliary variables = (1, xi1, xi2, …, xik):

with β = (βl,0, βl,1, βl,2, …, βl,k).

It is worth noting that P(yl,i = 1|xi) represents the odds of being food insecure. The natural log of the odds – also known as the logit – is as follows:

As a result, the logits are modeled with a multivariate linear regression:

where the error term εl,i takes only two values: εl,i = −βl when yl,i = 0. and εl,i = 1 − βl when yl,i = 1. Therefore, we cannot assume a normal distribution for the error term.

For both cases (yms,i, and ys,i), the multinomial regression model included both the levels of auxiliary variables identified as important by Boruta, and those representing important dimensions for data disaggregation (e.g. sex, age classes, income quintiles and geographic location of individuals). This was done in order to meet the condition of sample unbiasedness of the projection estimator in each disaggregation domain, as detailed in Section 3.

Table 4 presents results of the logistic regression model for the probability of being moderately or severely food insecure. Coefficients can be better interpreted in terms of eβ, which is the odds ratio (OR), holding all other variables constant. In this context, negative coefficients correspond to odds ratios lower than one, while positive coefficients to odds ratios greater than one.

As it can be seen, individuals belonging to the second and third level of education (between 9 and 15 years of education or with more than 15 years of education) are less likely to be moderately or severely food-insecure as shown by the negative signs of their coefficients. In fact, the odds of being severely or moderately food insecure of people in educat3 is 0.27 (=exp(-1.3122)) times that of individuals belonging to educat1.

From the same table, it is possible to observe that the probability of being moderately or severely food insecure is expected to decrease when income increases. Indeed, people in the fourth- and fifth-income quintiles show odds smaller than one.

Finally, another aspect that can be noted is that some of the variables identified as important by Boruta are not flagged as significant by the regression model. This is mainly due to the fact that Boruta also identifies non-linear relationships between variables, which are instead not considered by the adopted regression approach. This specific issue could be addressed in future extensions of this study.

Table 5 below presents results of the logistic regression model for the probability of being severely food insecure. As seen for prob.ms, individuals with higher education level (between 9 and 15 years of education or with more than 15 years of education) are less likely to be severely food insecure (negative signs of their estimated regression parameters). In this case, the odds have similar values to those found with the first model.

The effect of income is even clearer. Indeed, individuals falling in the higher income quintiles (Incat4 and Incat5) have a lower chance of being severely food insecure compared to individuals belonging to the first income quintile.

After fitting a regression model, a relevant issue is that of verifying model assumptions and performance. Obviously, methods selected to make this assessment will depend on the type of model at hand – e.g. ordinary regression, generalized linear regression, etc. This section presents some of the methods considered for the empirical exercise presented in the technical report, with the purpose of illustrating all the technical steps to implement the projection estimator.

A common approach adopted in this or similar contexts is the Hosmer and Lemeshow’s goodness of fit (GOF)[5], which deals with binary data. The model fits well when there is no significant difference between the model and the observed data (i.e. when the p-value is above 0.05). However, it is important to consider that most general methods to assess inference in case of independent and identically distributed (iid) variables (simple random sampling) can be misleading when applied to a sample obtained with stratified two-stage selection and unequal weighting of the units. Archer et al. (2007) demonstrate that standard goodness-of-fit tests are not always suitable for complex sample survey data, and propose alternative tests that account for complex design features, such as the F-adjusted mean residual test, which can be performed in Stata. The F-corrected Wald test was applied to both logistic regression models (implemented in the GWP dataset) leading to:

• A p-value close to 0 for the probability of being moderately or severely food-insecure, indicating a poor performance of the model;

• A p-value of 0.414 for the probability of being severely food-insecure, indicating that there is no statistical evidence to say that the model results in a poor fit.

As illustrated in previous sections, Boruta identified very few relevant auxiliary variables for the probability of being moderately or severely food-insecure, compared to the probability of being severely food-insecure. This may be an explanation of why the model poorly explains the distribution of the dependent variable. Furthermore, another aspect that could negatively affect the accuracy of the model is that variables identified as not important by Boruta, but needed for data disaggregation, have also been used to fit the regression to ensure unbiasedness. In general terms, having so few auxiliary variables available in the small sample (GWP) is certainly a limitation for the identification of a good model.

However, it should be stressed once again that this approach –being model assisted and not model based – is robust with respect to wrong model specifications.

4.9 Computing the synthetic values in the large sample

Having obtained the estimates of the parameters β, as illustrated in Section 4.8, the predicted probabilities are obtained as

The logit of the estimated probabilities P( = 1|xi) , with l = ms or l = s, can also be estimated with:

Which becomes:

for the probability of being moderately or severely food insecure, and

for the probability of being severely food insecure.[6]

Using the P( = 1|xi), values we can obtain the projection estimator:

for the total in the target population, and

for the proportion in the target population.

4.10 Disaggregated estimates and the assessment of their accuracy

Being indicator 2.1.2 obtained as realization of ratio-type estimators, the R function used to estimate the target parameter and its accuracy measures was svystatR() included in the package ReGenesees (Istat, Regenessees). This function allows producing estimates, standard errors and confidence intervals for ratio-type estimators, taking into account all relevant disaggregation dimensions (e.g by sex, age, income quintile and urban/rural location).

Variance estimation was performed by adopting the approach presented in Section 3 (see formula 3.3). The two components of the variance were estimated separately and then added together. Boxes 13 and 14 show how the function svystatR() was used to obtain disaggregated estimates and a measure of their accuracy for the two probability of interest.

Despite the non-optimal performance of the model, the use of the Horvitz-Thompson estimator guarantees robustness against its mis-specification. In other words, the HT estimator will be efficient if the specified model achieves a good fit, but maintains desirable properties such as design unbiasedness and design consistency even if the model is false.

Tables 6 and 7 present the disaggregated estimates along with their accuracy measures.

The comparison of projected and direct estimates in terms of their coefficient of variation (CV) shows that the former have greater or same accuracy than the latter in almost all cases. The only exception for both models is represented by the disaggregated estimate by geographic location (urban) and agecat_3 (50–64 years of age), where the CV of the indirect estimate is slightly higher than the one of the direct estimate. This was probably due to the fact that variables urban and agecat_3 were not important auxiliary variables in the two implemented regression models.

Similar steps to those illustrated in this section were implemented and comparable results were achieved using microdata from two additional countries – Guatemala and South Africa – proving the robustness of the proposed approach. The results of these two additional experiments are presented in separated annexes.

5. Conclusions and way forward

The indirect estimation approach presented in this report covers a great deal of interesting and relevant empirical applications for the production of disaggregated data for SDG (and other) indicators. In particular, most countries can normally rely on auxiliary variables provided by large-scale surveys, censuses, administrative records, or geospatial information. In this context, some of the target phenomena for SDG monitoring and data disaggregation are often too costly or complex to be incorporated in large-scale data collection campaigns. The presented approach allows measuring the variable of interest with a small-scale survey, on the sample of which the parameters of a regression-type statistical model can be estimated, by linking this variable to a set of auxiliary variables. Based on these parameters, the values of the target variable can be predicted on a larger-scale data source collecting the auxiliary information used to fit the model. Relying on a larger sample allows increasing the accuracy of disaggregated estimates and consider disaggregation domains that are not available in the small survey. In addition, predicting a variable of interest on the sample of a more extensive survey from which most national official statistics are produced, allows improving estimates’ consistency.

An additional aspect that should be highlighted is that the proposed strategy could be easily extended to other empirical contexts where, instead of integrating two independent surveys, the small survey could be integrated with auxiliary information coming from other types of data, such as censuses, administrative registers, and/or earth observation data. Furthermore, the extension of the projection estimator presented in section 3 of this report allows applying this approach to many other FAO-relevant SDG Indicators besides 2.1.2, such as:

1) SDG Indicator 2.1.1: Prevalence of Undernourishment;

2) SDG Indicator 2.3.1: Volume of production per labour unit by classes of farming/pastoral/forestry enterprise size;

3) SDG Indicator 2.3.2: Average income of small-scale food producers, by sex and indigenous status;

4) SDG Indicators 5.a.1.a (Percentage of people with ownership or secure rights over agricultural land (out of total agricultural population), by sex) and 5.a.1.b. (share of women among owners or rights-bearers of agricultural land, by type of tenure.

The three case studies presented in Section 4 and the Annexes considering SDG Indicator 2.1.2, show how – by using this approach - it is possible to increase the accuracy of disaggregated estimates in various disaggregation domains. However, for future extensions of the study or practical implementations in countries, the adoption of different methodological solutions will be considered. For example, a beta regression model could be tested instead of the logistic one. In addition, instead of modelling the probability of being moderately or severely food insecure, the prediction could be based on models fitted directly on the eight dichotomous variables collected with t

Annexes

Annex A: Projection estimator on microdata from Guatemala

This annex presents results based on the use of the projection estimator to produce disaggregated estimates of the probability of being moderately or severely food insecure on microdata from Guatemala.

The two surveys used to implement the proposed estimation approach are:

• Small sample: FIES individual module collected through the GWP. The Guatemala GWP dataset provides FIES data for a sample of 1 000 individuals divided into 125 enumeration areas. This dataset contains the same variables described in Table 2, with the exception of the variable on the employment and the marital status of respondents, and their ownership of a mobile phone. Given the availability of a larger-scale survey in the same year, the GWP dataset from 2014 was considered for the case study.

• Big sample: Encuesta Nacional de Condiciones de Vida (ENCOVI) 2014. The ENCOVI, implemented by the National Statistical Institute of Guatemala (INE) in 2014, collected information on 11 536 households and 35 069 individuals in a total of 1 037 primary sampling units.

Step 1: Recoding the variable of interest

Also in this case, the two probabilities of interest (prob.ms and prob.s) have been recoded into binary categorical values taking value 1 for probabilities higher than 0.5 and 0 otherwise.

Out of 1 000 sampling observations, 10 reported a missing value for both probabilities and were removed from the dataset.

Step 2: Recoding auxiliary variables

This second case study considered the same set of auxiliary variables included in the Malawi’s study (with the exception of marital and cellphone). These variables were recoded adopting the approach detailed in Section 4.5. For what concerns respondents’ education level, information collected with the Encuesta Nacional de Condiciones de Vida (ENCOVI) were recoded as illustrated below in Table 9.

Step 3: Selecting auxiliary variables to be included in the model

All auxiliary variables available in the GWP dataset were plugged into the Boruta algorithm to assess their relevance. The algorithm was implemented separately for the probability of being moderately or severely food insecure (Figure 8) and the probability of being severely food insecure (Figure 9). For brevity, in this annex we only report results of Boruta applied on dummies representing all levels of auxiliary variables.

Levels identified as important by Boruta:

• Inccat_1, Inccat_5 and Inccat_4 dummies.

• Educat_1, educat_2 and educat_3 dummies;

• Agecat_1 dummy;

• Rural dummy.

On the other hand, Boruta could not take a clear decision on agecat_4 and classified it as tentative.

The levels of auxiliary variables identified as important for the probability of being severely food insecure are:

• Inccat_5, Inccat_4, Inccat_2 dummies;

• sizeHH.

On the other hand, Boruta classified Inccat_3 and agecat_1 as tentative. Also in this case, the logistic regression models for the two probabilities of interest have been fitted using all levels of auxiliary variables identified as important or tentative by Boruta, along with all the relevant dimensions for data disaggregation.

Step 4: Estimating the projection parameters

The results of the two multinomial logistic regressions are reported respectively in Tables 10 and 11.

The goodness of fit, the F-adjusted mean residual test, was applied to both logistic regression models (implemented in the GWP dataset) leading to a p-value of 0.991 for the probability of being moderately or severely food-insecure, and 0 for the probability of being severely food-insecure. The low p-value obtained for the second model is justified by the fact that very few significant auxiliary variables were available.

Step 5: Producing disaggregated estimates and assessing their accuracy

Table 12 presents a comparison of direct estimates – obtained with the GWP dataset – and indirect estimates – obtained by using the projection estimator on the ENCOVI – for the prevalence of moderate or severe food insecurity in the population. Measures of accuracy for the two probabilities are provided, for different disaggregation levels.

As it can be seen from the table, indirect estimates are systematically more accurate than direct ones. In addition, in most cases, projected estimates are close to direct ones. The greatest difference between direct and indirect estimates can be observed for the urban/rural disaggregation.

Similarly, Table 13 presents a comparison of direct and indirect estimates – along with their accuracy measures - for the prevalence of severe food insecurity in the population.

One aspect that emerge from Tables 12 and 13 is that, while for the prevalence of moderate or severe food insecurity the CV of disaggregated estimates is always below 15 percent, the same cannot be said for the prevalence of severe food insecurity. The high variance of disaggregated estimates presented in Table 13 can be explained by the fact that, in the small sample, the variable of interest is very unbalanced (883 values equal to zero, 107 values equal to 1).

Annex B: Projection estimator on microdata from South Africa

This annex presents results from the application of the projection estimator to produce disaggregated estimates of the probability of being moderately or severely food insecure on microdata from South Africa. The two surveys used to implement the proposed approach are:

• Small Sample: FIES individual module collected through the GWP. The South Africa 2015 GWP dataset provides FIES data for a sample of 1 000 individuals divided into 125 enumeration areas. This dataset contains the same variables described in Table 2, with the exception of variables on the employment and marital status of respondents, and the ownership of a mobile phone.

• Big Sample: National income dynamics study (NIDS) 2015. The National income dynamics study is a panel survey implemented by the South Africa Labour and development research unit at the University of Cape Town. The survey was designed to provide information on poverty and well-being, at individual and household level, as well as wealth in terms of income and expenditures, demographic dynamics, education and employment. NIDS data are representative at the national level. The survey was first implemented in 2008 with a sample of over 28 000 individuals in 7 300 households across the country. The survey is repeated every two years with these same household members, who are called Continuing sample members (CSMs). The survey is designed to follow people who are CSMs, wherever they may be in South Africa at the time of interview. The NIDS data is therefore, by design, not representative provincially or at a lower level of territorial disaggregation.

Step 1: Recoding the variable of interest

The two probabilities (prob.ms and prob.s) have been recoded into binary categorical values taking value 1 for probabilities higher or equal to 0.5 and 0 otherwise.

Out of 1 000 sampling observations, 21 reported a missing value for both probabilities and were removed from the dataset.

Step 2: Recoding auxiliary variables

The same set of auxiliary variables used for Malawi was considered for the South African case study. These variables were recoded adopting the approach detailed in Section 4.5. For what concerns respondents’ education level, information collected with the NIDS were recoded as illustrated in Table 15.

Step 3: Selecting the auxiliary variables for the model

All the auxiliary variables available in the GWP dataset were plugged into the Boruta algorithm to assess their relevance. The algorithm was implemented separately for the probability of being moderately or severely food insecure (Figure 11) and the probability of being severely food insecure (Figure 12).

From Figure 11 it can be seen that levels identified as important by Boruta were:

• Inccat_5, Inccat_1, inccat_2 dummies;

• All levels of the education variable;

• Rural dummy.

Similarly, Figure 12 shows the important levels of auxiliary variables according to Boruta:

• Incat_5, Incat_1 and Incat_4 dummies.

• Educat_1 and educat_2 dummies.

• Agecat_3 dummy.

Step 4: Estimating the projection parameters

Two weighted multinomial logistic regressions were implemented to estimate the projection parameters to predict the values of the two variables of interest in the large survey (Tables 16 and 17).

The goodness of fit, the F-adjusted mean residual test, was applied to both logistic regression models leading to a p-value of 0.093 for the probability of being moderately or severely food-insecure, and 0 for the probability of being severely food-insecure.

Step 5: Producing disaggregated estimates and assessing their accuracy

After using the parameters estimated in Section B.5 to calculate the synthetic values of the two variables of interest in the large LSMS sample, estimates are produced alongside their coefficient of variation and confidence intervals considering all relevant disaggregation dimensions (e.g by sex, age_class, income quintile and urban/rural location).

Tables 18 and 19 show that most CVs of indirect estimates are lower than those of direct estimates.

References

Archer, K., Lemeshow, S. & Hosmer, D.W. 2007. Goodness-of-fit test for logistic regression models when data are collected using a complex sample design. Computational statistics & Data analysis, 51(9): 4450–4464. Breiman, L. 2001. Random forests. Machine learning, 45(1): 5–32 (also available at https://www.sciencedirect.com/science/article/pii/S0167947306002167)

Boruta. Boruta feature selection method of Kursa and Rudnicki (2010). [online] https://cran.r-project.org/web/packages/Boruta/Boruta.pdf [last accessed 2 February 2022]

Chambers, R.L. & Clark, R.G. 2015. An Introduction to Model-Based Sampling with Applications. Oxford Statistical Science Series No. 37. New York City, USA, Oxford University Press.

Cox, D.R. & Snell, E.J. 1989. Analysis of Binary Data. Second edition. London, Chapman & Hall/CRC Press.

FAO. 2022. [online] The Food Insecurity Experience Scale (FIES) webpage. https://www.fao.org/in-action/voices-of-the-hungry/fies/en/ (last accessed 2 February 2022).

FAO. 2021. Guidelines on data disaggregation for SDG Indicators using survey data. Rome. (also available at https://doi.org/10.4060/cb3253en)

Hansen, M.H., Madow, W.G. & Tepping, B.J. 1983. An Evaluation of Model Dependent and Probability-Sampling Inferences in Sample Survey (with discussion and rejoinder). Journal of the American Statistical Association 78. 776-807.

Harrell Jr., F.E. 2015. Describing, Resampling, Validating, and Simplifying the Model. In: Harrell Jr., F.E., Regression Modeling Strategies, Springer Series in Statistics. Switzerland, Springer International Publishing, pp. 103–126.

Italian National Institute of Statistics (Istat). Stat. Regenesees [online] https://www.istat.it/it/files//2021/09/ReGenesees.pdf (last accessed 2 February 2022)

Kim, J.K. & Rao, J.N.K. 2012. Combining data from two independent surveys: a model-assisted approach. Biometrika, 99(1): 85–100.

Kursa, M. & Rudnicki, W., 2010. Feature Selection with the Boruta Package. Journal of Statistical Software. September 2010. Volume 36, Issue 11. https://www.jstatsoft.org/article/view/v036i11

Manyamba, C. 2013. Voices of the Hungry Project – Piloting the Global Food Insecurity Experience Scale for the Gallup World Poll in Malawi: Linguistic adaptation in Chichewa and Chitumbuka. Rome, FAO. (available at http://www.fao.org/fileadmin/templates/ess/voh/MALAWI_FIES_Language_Adaptation_Report_Aug2013.pdf).

McFadden, D. 1974. Conditional logit analysis of qualitative choice behavior. In P. Zarembka, ed., Frontiers in Econometrics, pp. 105–142. New York City, USA, Academic Press.

Nagelkerke, N.J.D. 1991. A Note on a General Definition of the Coefficient of Determination. Biometrika, 78, 691-692. https://doi.org/10.1093/biomet/78.3.691

National Statistical Office (Malawi). 2017. Malawi Integrated Household Panel Survey (IHPS) 2016 – Basic Information Document. National Statistical Office of Malawi. https://microdata.worldbank.org/index.php/catalog/2939/download/48116

Royall, R.M. 1976. The linear least squared prediction to two stage sampling. Journal of The American statistical Association. 71. 657-674.

Ryan, T.P. 2008. Modern Regression Methods, Second edition. Wiley Series in Probability and Statistics. New York City, USA, John Wiley & Sons Book Series.

Särndal, C.-E., Swensson, B. & Wretman, J. 1992. Model Assisted Survey Sampling. New York City, USA, Springer-Verlag.

Singh, C.A. & Mohl, A.C. 1996. Understanding Calibration Estimators in Survey Sampling. Survey Methodology.22(2): 107–115.

Smith, M.D., Rabbitt, M.P. & Coleman-Jensen, A. 2017. Who are the World’s Food Insecure? New Evidence from the Food and Agriculture Organization’s Food Insecurity Experience Scale. World Development, 93: 402–412.

Tillé, Y. 2019. Sampling and Estimation from Finite Populations. Wiley.

United Nations Statistical Division. 2022. [online]. Data Disaggregation for the SDG Indicators website. https://unstats.un.org/sdgs/iaeg-sdgs/disaggregation/ (last accessed: 2 February 2022).

Valliant, R., Dorfmann, A.H & Royall, R.M. 2000. Finite Population Sampling and Inference: A Predicition Approach. New York City, USA, John Wiley & Sons.

Woodruff, R.S. 1971. A Simple Method for Approximating the Variance of a Complicated Estimate. Journal of the American Statistical Association. 66(334): 411–414.

World Bank Microdata Library. (Fourth Integrated Household Survey 2016-2017) [online]. https://microdata.worldbank.org/index.php/catalog/2936 (last accessed 2 February 2022).

Zardetto D. 2015. “ReGenesees: an Advanced R System for Calibration, Estimation and Sampling Error Assessment in Complex Sample Surveys” (extended version). Journal of Official Statistics, 31(2):177-203.

1 In the relevant literature, “small area” is intended as a general concept, and is used to indicate a general partition of the population according to geographical criteria or other structural characteristics (e.g. sociodemographic variables for household surveys or economic variables for business surveys).

2 An ancillary statistics is a measure extracted from a sample, the result of which does not depend on the parameter(s) of a model.

3 More details on the mandatory and future dimensions for data disaggregation of SDG Indicator 2.1.2, the authors refer to the compilation of categories and dimensions of data disaggregation prepared by the IAEG-DGs (UNSD, 2022).

4 The territory of Malawi is divided in regions (central, northern, and southern), which are in turn divided in a total of 28 districts.

5 In addition to the goodness of fit, measures of the extent of variation explained by the model could be considered to assess its performance. In case of logistic regressions, measure of pseudo R-squares – especially McFadden (1974), Cox and Snell (1989) and Nagelkerke (1991) – are possible alternatives. The psrsq() function in R produces the Nagelkerke and Cox–Snell pseudo R-square estimates for survey sample data.

6 For the definitions of dummies used as auxiliary variables, the reader should refer to Section 4.5 of this technical report.

7 The initial category “Other” was mainly selected by individuals with disabilities attending special trainings.