Automated Data Observatories

How We Add Value to Public Data With Imputation and Forecasting

Mon, 08 Nov 2021 10:00:00 +0100

Public data sources are often plagued by missng values. Naively you may think that you can ignore them, but think twice: in most cases, missing data in a table is not missing information, but rather malformatted information. This approach of ignoring or dropping missing values will not be feasible or robust when you want to make a beautiful visualization, or use data in a business forecasting model, a machine learning (AI) applicaton, or a more complex scientific model. All of the above require complete datasets, and naively discarding missing data points amounts to an excessive waste of information. In this example we are continuing the example a not-so-easy to find public dataset.

In the previous blogpost we explained how we added value by documenting data following the FAIR principle and with the professional curatorial work of placing the data in context, and linking it to other information sources, such as other datasets, books, and publications, regardless of their natural language (i.e., whether these sources are described in English, German, Portugese or Croatian). Photo: Jack Sloop.

Completing missing datapoints requires statistical production information (why might the data be missing?) and data science knowhow (how to impute the missing value.) If you do not have a good statistician or data scientist in your team, you will need high-quality, complete datasets. This is what our automated data observatories provide.

Why is data missing?

International organizations offer many statistical products, but usually they are on an ‘as-is’ basis. For example, Eurostat is the world’s premiere statistical agency, but it has no right to overrule whatever data the member states of the European Union, and some other cooperating European countries give to them. And they cannot force these countries to hand over data if they fail to do so. As a result, there will be many data points that are missing, and often data points that have wrong (obsolete) descriptions or geographical dimensions. We will show the geographical aspect of the problem in a separate blogpost; for now, we only focus on missing data.

Some countries have only recently started providing data to the Eurostat umbrella organization, and it is likely that you will find few datapoints for North Macedonia or Bosnia-Herzegovina. Other countries provide data with some delay, and the last one or two years are missing. And there are gaps in some countries’ data, too.

See the authoritative copy of the dataset.

This is a headache if you want to use the data in some machine learning application or in a multiple or panel regression model. You can, of course, discard countries or years where you do not have full data coverage, but this approach usually wastes too much information–if you work with 12 years, and only one data point is available, you would be discarding an entire country’s 11-years’ worth of data. Another option is to estimate the values, or otherwise impute the missing data, when this is possible with reasonable precision. This is where things get tricky, and you will likely need a statistician or a data scientist onboard.

What can we improve?

Consider that the data is only missing from one year for a particular country, 2015. The naive solution would be to omit 2015 or the country at hand from the dataset. This is pretty destructive, because we know a lot about the radio market turnover in this country and in this year! But leaving 2015 blank will not look good on a chart, and will make your machine learning application or your regression model stop.

A statistician or a radio market expert will tell you that you know more-or-less the missing information: the total turnover was certainly not zero in that year. With some statistical or radio domain-specific knowledge you will use the 2014, or 2016 value, or a combination of the two and keep the country and year in the dataset.

Our improved dataset added backcasted (using the best time series model fitting the country’s actually present data), forecasted (again, using the best time series model), and approximated data (using linear approximation.) In a few cases, we add the last or next known value. To give a few quantiative indicators about our work:

Increased number of observations: 65%
Reduced missing values: -48.1%
Increased non-missing subset for regression or AI: +66.67%

If your organization is working with panel (longitudional multiple) regressions or various machine learning applications, then your team knows that not havint the +66.67% gain would be a deal-breaker in the choice of models and punctuality of estimates or KPIs or other quantiative products. And that they would spent about 90% of their data resources on achieving this +66.67% gain in usability.

If you happen to work in an NGO, a business unit or a research institute that does not employ data scientists, then it is likely that you can never achieve this improvement, and you have to give up on a number of quantitative tools or visualizations. If you have a data scientist onboard, that professional can use our work as a starting point.

Can you trust our data?

We believe that you can trust our data better than the original public source. We use statistical expertise to find out why data may be missing. Often, it is present in a wrong location (for example, the name of a region changed.)

If you are reluctant to use estimates, think about discarding known actual data from your forecast or visualization, because one data point is missing. How do you provide more accurate information? By hiding known actual data, because one point is missing, or by using all known data and an estimate?

Our codebooks and our API uses the Statistical Data and Metadata eXchange documentation standards to clearly indicate which data is observed, which is missing, which is estimated, and of course, also how it is estimated. This example highlights another important aspect of data trustworthiness. If you have a better idea, you can replace them with a better estimate.

Our indicators come with standardized codebooks that do not only contain the descriptive metadata, but administrative metadata about the history of the indicator values. You will find very important information about the statistical method we used the fill in the data gaps, and even link the reliable, the peer-reviewed scientific, statistical software that made the calculations. For data scientists, we record the plenty of information about the computing environment, too-–this can come handy if your estimates need external authentication, or you suspect a bug.

Avoid the data Sisyphus

If you work in an academic institution, in an NGO or a consultancy, you can never be sure who downloaded the Annual detailed enterprise statistics for services (NACE Rev. 2 H-N and S95) Eurostat folder from Eurostat. Did they modify the dataset? Did they already make corrections with the missing data? What method did they use? To prevent many potential problems, you will likely download it again, and again, and again…

See our The Data Sisyphus blogpost.

We have a better solution. You can always rely on our API to import directly the latest, best data, but if you want to be sure, you can use our regular backups on Zenodo. Zenodo is an open science repository managed by CERN and supported by the European Union. On Zenodo, you can find an authoritative copy of our indicator (and its previous versions) with a digital object identifier, in this case, 10.5281/zenodo.5652118. These datasets will be preserved for decades, and nobody can manipulate them. You cannot accidentally overwrite them, and we have no backdoor access to modify them.

Are you a data user? Give us some feedback! Shall we do some further automatic data enhancements with our datasets? Document with different metadata? Link more information for business, policy, or academic use? Please give us any feedback!

How We Add Value to Public Data With Better Curation And Documentation?

Mon, 08 Nov 2021 09:00:00 +0100

In this example, we show a simple indicator: the Turnover in Radio Broadcasting Enterprises in many European countries. This is an important demand driver in the Music economy pillar of our Digital Music Observatory, and important indicator in our more general Cultural & Creative Sectors and Industries Observatory. We show a very similar example in our Green Deal Data Observatory with environmental R&D public spending in Europe.

This dataset comes from a public datasource, the data warehouse of the European statistical agency, Eurostat. Yet it is not trivial to use: unless you are familiar with national accounts, you will not find this dataset on the Eurostat website.

The data can be retrieved from the Annual detailed enterprise statistics for services NACE Rev.2 H-N and S95 Eurostat folder.

Our version of this statistical indicator is documented following the FAIR principles: our data assets are findable, accessible, interoperable, and reusable. While the Eurostat data warehouse partly fulfills these important data quality expectations, we can improve them significantly. And we can also improve the dataset, too, as we will show in the next blogpost.

Findable Data

Our data observatories add value by curating the data–we bring this indicator to light with a more descriptive name, and we place it in context with our Digital Music Observatory and Cultural & Creative Sectors and Industries Observatory. While many people may need this dataset in the creative sectors, or among cultural policy designers, most of them have no training in working with national accounts, which imply decyphering national account data codes in records that measure economic activity at a national level. Our curated data observatories bring together many available data around important domains. Our Digital Music Observatory, for example, aims to form an ecosystem of music data users and producers.

We added descriptive metadata that help you find our data and match it with other relevant data sources.

We added descriptive metadata that help you find our data and match it with other relevant data sources. For example, we add keywords and standardized metadata identifiers from the Library of Congress Linked Data Services, probably the world’s largest standardized knowledge library description. This ensures that you can find relevant data around the same key term (radio broadcasting) in addition to our turnover data. This allows connecting our dataset unambiguosly with other information sources that use the same concept, but may be listed under different keywords, such as Radio–Broadcasting, or Radio industry and trade, or maybe Hörfunkveranstalter in German, or Emitiranje radijskog programa in Croatian or Actividades de radiodifusão in Portugese.

Accessible Data

Our data is accessible in two forms: in csv tabular format (which can be read with Excel, OpenOffice, Numbers, SPSS and many similar spreadsheet or statistical applications) and in JSON for automated importing into your databases. We can also provide our users with SQLite databases, which are fully functional, single user relational databases.

Tidy datasets are easy to manipulate, model and visualize, and have a specific structure: each variable is a column, each observation is a row, and each type of observational unit is a table. This makes the data easier to clean, and far more easier to use in a much wider range of applications than the original data we used. In theory, this is a simple objective, yet we find that even governmental statistical agencies–and even scientific publications–often publish untidy data. This poses a significant problem that implies productivity loses: tidying data will require long hours of investment, and if a reproducible workflow is not used, data integrity can also be compromised: chances are that the process of tidying will overwrite, delete, or omit a data or a label.

Tidy datasets are easy to manipulate, model and visualize, and have a specific structure: each variable is a column, each observation is a row, and each type of observational unit is a table.

While the original data source, the Eurostat data warehouse is accessible, too, we added value with bringing the data into a tidy format. Tidy data can immediately be imported into a statistical application like SPSS or STATA, or into your own database. It is immediately available for plotting in Excel, OpenOffice or Numbers.

Interoperability

Our data can be easily imported with, or joined with data from other internal or external sources.

All our indicators come with standardized descriptive metadata, and statistical (processing) metadata. See our API

All our indicators come with standardized descriptive metadata, following two important standards, the Dublin Core and DataCite–implementing not only the mandatory, but the recommended descriptions, too. This will make it far easier to connect the data with other data sources, e.g. turnover with the number of radio broadcasting enterprises or radio stations within specific territories.

Our passion for documentation standards and best practices goes much further: our data uses Statistical Data and Metadata eXchange standardized codebooks, unit descriptions and other statistical and administrative metadata.

Reuse

All our datasets come with standardized information about reusabililty. We add citation, attribution data, and licensing terms. Most of our datasets can be used without commercial restriction after acknowledging the source, but we sometimes work with less permissible data licenses.

In the case presented here, we added further value to encourage re-use. In addition to tidying, we significantly increased the usability of public data by handling missing cases. This is the subject of our next blogpost.

Digital Music Observatory on MaMA 2021

Thu, 14 Oct 2021 12:15:00 +0000

Use Cases

Public advocacy reports, scientific uses

Use Cases 2

Business Confidential Reports with Digital Music Observatory

Damage claims in private copying
Royalty setting for restaurants, hotels, broadcasting
Music streaming market indicators
Evidence for competition law / regulatory affairs

Questions?

LinkedIn: Daniel Antal - Digital Music Observatory

Open Data - The New Gold Without the Rush

Thu, 14 Oct 2021 12:15:00 +0000

Use Cases

Public advocacy reports, scientific uses

Use Cases 2

Business Confidential Reports with Digital Music Observatory

Damage claims in private copying
Royalty setting for restaurants, hotels, broadcasting
Music streaming market indicators
Evidence for competition law / regulatory affairs

Questions?

LinkedIn: Daniel Antal - Digital Music Observatory

Digital Music Observatory on the MaMA Convention 2021

Thu, 14 Oct 2021 11:00:00 +0200

Currently more than half of the global music sales are made by autonomous AI systems owned by Google, Apple, or Spotify. These data monopolies are getting rich, because they reap the profit from music businesses with an average employee count of 1.8 Europe. European music businesses are easy to exploit with armies of data engineers and data scientists because they do not have a single data scientist or even an IT function.

Artists in the UK had a difficulty explaining in Westminster how they are losing out in streaming– so we have created a streaming price index, like the Dow Jones, if you like, that explains the economic factors of the devaluation of music in the last 5 years in 20 countries. (See our report.)
Music organizations in Slovakia and Hungary were frustrated that their politicians and journalists believed music to be taxpayer funded, so we showed with data that they contribute more proportionally to the national budget than car manufacturers, the darling of local politicians (See our reports in Hungary (recast several times) and in Slovakia.)
We successfully challenged with data restaurant associations, hotel chains, telecom corporations and broadcasters who wanted to bring music prices down in court and via lobbying.

The music industry has envied the television and film industry which has a single go-to-point for data when it needs them, the European Audiovisual Observatory. It started lobbying for a publicly financed music observatory. But we did not wait. The music industry has a tragic track record of failed centralized international data projects. We built Reprex out of a 12-country, decentralized music project. We learned how to utilize hidden, but already existing data and research funds well, and how to manage the data governance among the poisonous conflicts of interests between rich and poor countries, authors vs producers, producer’s vs performers.

Our Digital Music Observatory is not theoretical, it is practical, because it is built around real-life court cases, damage claims, lobbying and PR arguments.
Our Digital Music Observatory is comprehensive – it contains more than a thousand indicators from all European countries. We have enough data to test the biases of the Spotify or the YouTube algorithm – you would be surprised what the data tells us.
It has data available much sooner, in much higher quality and in a more practical format than in the Audiovisual one.

Presentation Slides

You can see the presentation slides here.

Crunchconf: Open Data, New Gold Without the Rush

Fri, 08 Oct 2021 10:10:00 +0200

Every year, the EU announces that billions and billions of data are now “open” again, but this is not gold. At least not in the form of nicely minted gold coins, but in gold dust and nuggets found in the muddy banks of chilly rivers. There is no rush for it, because panning out its value requires a lot of hours of hard work. Our goal is to automate this work to make open data usable at scale, even in trustworthy AI solutions.

Summary

In his presentation, Daniel compared the current state of open data (including governmental open data and scientific open data) to a thrift store. You can often find bargains, or historical data that would be impossible to source from data vendors, but on a strictly as-is basis, without a catalogue, service, or guarantee. Therefore, working with open data requires a careful reprocessing, validation, and in many cases, frequent re-validation. Open data is often over-estimated: it is never a finished product, often it cannot even be downloaded, therefore it requires further investment to make it valuable. However, because most open data arrives from the governmental sector, you can tap into information sources where no market alternative exists. Open data in some cases may be a cheaper substitute to market vendors, but often it is an exclusive source of information that do not have any market vendors.

Sisyphus was punished by being forced to roll an immense boulder up a hill only for it to roll down every time it neared the top, repeating this action for eternity. This is the price that project managers and analysts pay for the inadequate documentation of their data assets.

The practices related to the exploitation of open data are not only relevant in an open data context: these are good data ingestion and procurement practices for any third party data, and in large organizations, for any cross-departmental data. (See the blogpost: The Data Sisyphus.)
Case Study: Belgian Drought/Flood Risk Awareness, Financial Capacity & Hydrology a complex integration of various open data sources.

In the second part of the presentation, Daniel talked about our modern data observatory concept. We have reviewed about 80 functioning and already defunct international data collection programs. Data observatories, like Copernicus’ Observatory, are permanent infrastructure to record various domain-specific data, such as alternative fuel information, information on homelessness, or on the European music business. In our assessment, most of the EU, OECD, UNESCO recognized or endorsed observatories use obsolete technology and do not rely on the new achievements of data science. Reprex, our start-up offers an open source, open data based alternative solution to build largely automated data observatories. We believe that human judgement is needed in data curation, but processing, documentation and validation is best done by computers.

Case Study: Reprocessing geographical information with administrative boundary changes

At last, he presented a few development directions with our open-source software, mentioning our work withing the rOpenGov community. This part of the presentation was originally meant to open the way for a half-day open data workshop, but due to the current pandemic situation, the physical part of the conference and the workshops were not held.

The presentation largely included the topics of our Data & Lyrics blogpost: Open Data—The New Gold Without the Rush

Presentation Slides

See the presentation slides here.

CCSI Data Observatory

Wed, 06 Oct 2021 16:00:00 +0200

The creative and cultural sectors and industries are mainly made of networks of freelancers and microenterprises, with very few medium-sized companies. Their economic performance, problems, and innovation capacities hidden. Our open collaboration to create this data observatory is committed to change this. Relying on modern data science, the re-use of open governmental data, open science data, and novel harmonized data collection we aim to fill in the gaps left in the official statistics of the European Union.

We believe that introducing Open Policy Analysis standards with open data, open-source software and research automation can help better understanding how creative people and their enterprises and institutions add value to the European economy, how they create jobs, innovate, and increase the well-being of a diverse European society. Our collaboration is open for individuals, citizens scientists.

The new observatory can be reached on ccsi.dataobservatory.eu and will be institutionally hosted by IViR, the Institute for Information Law of the University of Amsterdam, where Reprex’s co-founder, Daniel Antal will coordinate the development of this new, open scientific tool. Reprex will continue to develop the working model of the data observatory and continue to build open source software tools within the rOpenGov community and the R-Universe initative of ROpenSci.

The Scuola Superiore di Studi Universitari e di Perfezionamento Sant’Anna and Università degli Studi di Trento (Italy); University of Glasgow (United Kingdom); Universiteit van Amsterdam and Stichting Europeana from the Netherlands; the National University of Ireland Maynooth (Ireland); Tartu Ulikool (Estonia); Szegedi Tudományegyetem (Hungary); Fundacion Santa Maria La Real del Patrimonio Historico from Spain; the Katholieke Universiteit Leuven, (Belgium); Culture Action Europe AISBL and IDEA Strategische Economische Consulting (Belgium) and Reprex created the the RECREO consortium, which will mainly develop new policy evidence in the field of innovation and inclusiveness for the creative and cultural sectors, industries. The Consortium applies for a Horizon Europe grant with the HORIZON-CL2-2021-HERITAGE-01-03 Cultural and creative industries as a driver of innovation and competitiveness call of the European Commission.

Creative & Cultural Sectors Industries Data Observatory

Tue, 05 Oct 2021 18:00:00 +0000

The Data Sisyphus

Thu, 08 Jul 2021 09:00:00 +0200

When was a file downloaded from the internet? What happened with it sense? Are their updates? Did the bibliographical reference was made for quotations? Missing values imputed? Currency translated? Who knows about it – who created a dataset, who contributed to it? Which is an intermediate format of a spreadsheet file, and which is the final, checked, approved by a senior manager?

Big data creates inequality and injustice. On aspect of this inequality is the cost of data processing and documentation – a greatly underestimated, and usually not reported cost item. In small organizations, where there are no separate data science and data engineering roles, data is usually supposed to be processed and documented by (junior) analysts or researchers. This a very important source of the gap between Big Tech and them: the data usually ends up very expensive, ill-formatted, not readable by computers that use machine learning and AI. Usually the documentation steps are completely omitted.

“Data is potential information, analogous to potential energy: work is required to release it.” – Jeffrey Pomerantz

Metadata, which is information about the history of the data, and information how it can be technically and legally reused, has a hidden cost. Cheap or low-quality external data comes with poor or no metadata, and small organizations lack the resources to add high-quality metadata to their datasets. However, this only perpetuates the problem.

The hidden cost item behind the unbillable hours

As we have shown with our research partners, such metadata problems are not unique to data analysis. Independent artists and small labels are suffering on music or book sales platforms, because their copyrighted content is not well documented. If you automatically document tens of thousands of songs or datasets, the documentation cost is very small per item. If you, do it manually, the cost may be higher than the expected revenue from the song, or the total cost of the dataset itself. (See our research consortiums' preprint paper: Ensuring the Visibility and Accessibility of European Creative Content on the World Market: The Need for Copyright Data Improvement in the Light of New Technologies)

In the short run, small consultancies, NGOs, or as a matter of fact, musicians, seem to logically give up on high-quality documentation and logging. In the long run, this has two devastating consequences: computers, such as machine learning algorithms cannot read their documents, data, songs. And as memory fades, the ill-documented resources need to be re-created, re-checked, reformatted. Often, they are even hard to find on your internal server or laptop archive.

Metadata is a hidden destroyer of the competitiveness of corporate or academic research, or independent content management. It never quoted on external data vendor invoices, it is not planned as a cost item, because metadata, the description of a dataset, a document, a presentation, or song, is meaningless without the resource that it describes. You never buy metadata. But if your dataset comes without proper metadata documentation, you are bound, like Sisyphus, to search for it, to re-arrange it, to check its currency units, its digits, its formatting. Data analysts are reported to spend about 80% of their working hours on data processing and not data analysis – partly, because data processing is a very laborious task that can be done by computers at a scale far cheaper, and partly because they do not know if the person who sat before them at the same desk has already performed these tasks, or if the person responsible for quality control checked for errors.

Uncut diamonds need to be cut, polished, and you have to make sure that they come from a legal source. Data is similar: it needs to be tidied up, checked and documented before use. Photo: Dave Fischer.

Undocumented data is hardly informative – it may be a page in a book, a file in an obsolete file format on a governmental server, an Excel sheet that you do not remember to have checked for updates. Most data are useless, because we do not know how it can inform us, or we do not know if we can trust it. The processing can be a daunting task, not to mention the most boring and often neglected documentation duties after the dataset is final and pronounced error-free by the person in charge of quality control.

Our observatory automatically processes and documents the data

The good news about documentation and data validation costs is that they can be shared. If many users need GDP/capita data from all over the world in euros, then it is enough if only one entity, a data observatory, collects all GDP and population data expresed in dollars, korunas, and euros, and makes sure that the latest data is correctly translated to euros, and then correctly divided by the latest population figures. These task are error-prone,and should not be repeaeted by every data journalist, NGO employee, PhD student or junior analyst. This is one of the services of our data observatory.

The tidy data format means that the data has a uniform and clear data structure and semantics, therefore it can be automatically validated for many common errors and can be automatically documented by either our software or any other professional data science application. It is not as strict as the schema for a relational database, but it is strict enough to make, among other things, importing into a database easy.
The descriptive metadata contains information on how to find the data, access the data, join it with other data (interoperability) and use it, and reuse it, even years from now. Among others, it contains file format information and intellectual property rights information.
The processing metadata makes the data usable in strictly regulated professional environments, such as in public administration, law firms, investment consultancies, or in scientific research. We give you the entire processing history of the data, which makes peer-review or external audit much easier and cheaper.
The authoritative copy is held at an independent repository, it has a globally unique identifier that protects you from accidental data loss, mixing up with unfinished an untested version.

Cutting the dataset to a format with clear semantics and documenting it with the FAIR metadata concep exponentially increases the value of data. It can be publisehd or sold at a premium. Photo: Andere Andre.

While humans are much better at analysing the information and human agency is required for trustworthy AI, computers are much better at processing and documenting data. We apply to important concepts to our data service: we always process the data to the tidy format, we create an authoritative copy, and we always automatically add descriptive and processing metadata.

The value of metadata

Metadata is often more valuable and more costly to make than the data itself, yet it remains an elusive concept for senior or financial management. Metadata is information about how to correctly use the data and has no value without the data itself. Data acquisition, such as buying from a data vendor, or paying an opinion polling company, or external data consultants appears among the material costs, but metadata is never sold alone, and you do not see its cost.

In most cases, the reason why there is no gold rush for open data is that fact that while the EU member states release billions of euros' worth data for free, or at very low cost, annually, it comes without proper metadata.

Data-as-Service

Reusable, legal, easy-to-import, interoperable, always fresh data in tidy formats with a modern API. Photo: Edgar Soto.

If the data source is cheap or has a low quality, you do not even get it. If you do not have it, it will show up as a human resource cost in research (when your analysist or junior researcher are spending countless hours to find out the missing metadata information on the correct use of the data) or in sales costs (when you try to reuse a research, consulting or legal product and you have comb through your archive and retest elements again and again.)

The data, together with the descriptive and administrative metadata, and links to the use license and the authoritative copy can be found in our API. Try it out!

Green Deal Data Observatory

Wed, 07 Jul 2021 00:00:00 +0000

Finding reliable historic and new data and information about climate change, as well as the impact of various European Green Deal policies that try to mitigate it is surprisingly hard to find if you are a scientific researcher. And it is even more hopeless if you work as a (data) journalist, a policy researcher in an NGO, or in the sustainability unit of a company that does not provide you with an army of (geo)statisticians, data engineers, and data scientists who can render various data into usable format, i.e.something that you can trust, quote, visualize, import, or copy & paste.

Visit the Green Deal Data Observatory

Try the Green Deal Data Observatory API

Connect on LinkedIn

Better, Bigger, Faster, More

Novel data products	Official statistics at the national and European levels follow legal regulations, and in the EU, compromises between member states. New policy indicators often appear 5-10 years after demand appears. We employ the same methodology, software, and often even the same data that Eurostat might use to develop policy indicators, but we do not have to wait for a political and legal consensus to create new datasets. See our 100,000 Opinions on the Most Pressing Global Problem blogpost.
Better data	Statistical agencies, old fashioned observatories, and data providers often do not have the mandate, know-how or resources to improve data quality. Using peer-reviewed statistical software and hundreds of computational tests, we are able to correct mistakes, impute missing data, generate forecasts, and increase the information content of public data by 20-200% percent. This makes the data usable for NGOs, journalists, and visual artists—among other potential users—who do not have this statistical know-how to make incomplete, mislabelled or low quality data usable for their needs and applications. See our example with the Government Budget Allocations for R&D in Environment indicator.
Never seen data	The 2019/1024 directive on open data and the re-use of public sector information of the European Union (which is an extension and modernization of the earlier directives on re-use of public sector information since 2003) makes data gathered in EU institutions, national institutions, and municipalities, as well as state-owned companies legally available. According to the European Data Portal the estimated historical cost of the data released annually is in the billions of euros. But if this data is a gold mine, its full potential can only be unlocked by an experienced data mining partner like Reprex. Here is why: data is not readily downloadable; it sits in various obsolete file formats in disorganized databases; it is documented in various languages, or not documented at all; it is plagued with various processing errors. We make the powerful promise of open data of the EU legislation a reality in the field of the Green Deal policy context.

Increase Your Impact, Avoid Old Mistakes

Reprex helps its policy, business, and scientific partners by providing efficient solutions for necessary data engineering, data processing and statistical tasks that are as complex as they are tedious to perform. We deploy validated, open-source, peer-reviewed scientific software to create up-to-date, reliable, high-quality, and immediately usable data and visualizations. Our partners can leave the burden of this task, share the cost of data processing, and concentrate on what they do best: disseminating and advocating, researching, or setting sustainable business or underwriting indicators and creating early warning systems.

Impact

We publish the data in a way that it is easy to find—as a separate data publication with a DOI, full library metadata, and place it in open science repositories. Our data is more findable than 99% of the open science data, and therefore makes far bigger impact. See our data on the European open science repository Zenodo managed by CERN (the European Organization for Nuclear Research).

Easy-to-use data

Our data follows the tidy data principle and comes with all the recommended Dublin Core and DataCite metadata. This increases our data compatibility, allowing users to open it in any spreadsheet application or import into their databases. We publish the data in tabular form, and in JSON form through our API enabling automatic retrieval for heavy users, especially if they plan to automatically use our data in daily or weekly updates. Using the best practice of data formatting and documentation with metadata ensures reproducibility and data integrity, rather than repeating data processing and preparation steps (e.g. changing data formats, removing unwanted characters, creating documentation, and other data processing steps that take up thousands of working hours. See our blogpost on the data Sisyphus.

Ethical Big Data for All

Big data creates inequalities, because only the largest corporations, government bureaucracies and best endowed universities can afford large data collection programs, the use of satellites, and the employment of many data scientists. Our open collaboration method of data pooling and cost sharing makes big data available for all.

Big picture

Integrating and joining data is hard—it requires engineering, mathematical, and geo-statistical know-how that a large amount of environmental users and stakeholders do not possess. Some examples of the challenges implicit in making data usable include addressing the changing boundaries of French departments (and European administrative-geographic borders, in general), various projections of coordinates on satellite images of land cover, different measurement areas for public opinion and hydrological data, public finance expressed in different orders (e.g. millions versus thousands of euros). We create data that is easy to combine, map, and visualize for end users. See our case study on the severity and awareness of flood risk in Belgium, as well as the financial capacity to manage it.

Ethical, Trustworthy AI

AI in 2021 increases data inequalities because large government and corporate entities with an army of data engineers can create proprietary, black box business algorithms that fundamentally alter our lives. We are involved in the R&D and advocacy of the EU’s trustworthy AI agenda which aims at similar protections like GDPR in privacy. We want to demystify AI by making it available for organizations who cannot finance a data engineering team, because 95% of a successful AI is cheap, complete, reliable data tested for negative outcomes – precisely what dwe offer to our users.

Open Collaboration

Reprex grew out of an international data cooperation and works in the open-source world. We use the agile open collaboration method that allows us to work with large corporations, NGOs, developers, university researcher institutes and individuals on an equal footing.

Find us on LinkedIn or send us an email.

Metadata

Wed, 07 Jul 2021 00:00:00 +0000

Adding metadata exponentially increases the value of data. Did your region add a new town to its boundaries? How do you adjust old data to conform to constantly changing geographic boundaries? What are some practical ways of combining satellite sensory data with my organization’s records? And do I have the right to do so? Metadata logs the history of data, providing instructions on how to reuse it, also setting the terms of use. We automate this labor-intensive process applying the FAIR data concept.

In our observatory we apply the concept of FAIR (findable, accessibe, interoperable, and reusable digital assets) in our APIs and in our open-source statistical software packages.

The hidden cost item

Metadata gets less attention than data, because it is never acquired separately, it is not on the invoice, and therefore it remains an a hidden cost, and it is more important from a budgeting and a usability point of view than the data itself. Metadata is responsible for industry non-billable hours or uncredited working hours in academia. Poor data documentation, lack of reproducible processing and testing logs, inconsistent use of currencies, keywords, and storing messy data make reusability and interoperability, integration with other information impossible.

FAIR Data and the Added Value of Rich Metadata we introduce how we apply the concept of FAIR (findable, accessibe, interoperable, and reusable digital assets) in our APIs.

Organizations pay many times for the same, repeated work, because these boring tasks, which often comprise of tens of thousands of microtasks, are neglected. Our solution creates automatic documentation and metadata for your own historical internal data or for acquisitions from data vendors. We apply the more general Dublin Core and the more specific, mandatory and recommended values of DataCite for datasets – these are new requirements in EU-funded research from 2021. But they are just the minimal steps, and there is a lot more to do to create a diamond ring from an uncut gem.

Map your data: bibliographis, catalogues, codebooks, versioning

Updating descriptive metadata, such as bibliographic citation files, descriptions and sources to data files downloaded from the internet, versioning spreadsheet documents and presentations is usually a hated and often neglected task withing organization, and rightly so: these boring and error-prone tasks are best left to computers.

Already adjusted spreadsheets are re-adjusted and re-checked. Hours are spent on looking for the right document with the rigth version. Duplicates multiply. Already downloaded data is downloaded again, and miscategorized, again. Finding the data without map is a treasure hunt. Photo: © N.

The lack of time and resources spend on documentation over time reduces reusability and significantly increases data processing and supervision or auditing costs.

Our observatory metadata is compliant with the Dublin Core Cross-Domain Attribute Set metadata standard, but we use different formatting. We offer simple re-formatting from the richer DataCite to Dublin Core for interoperability with a wider set of data sources.
We use all mandatory DataCite metadata fields, all the the recommended and optional ones.
It complies with the tidy data principles.

In other words: very easy to import into your databases, or join with other databases, and the information is easy to find. Corrections, updates can automatically managed.

What happened with the data before?

We are creating Codebooks that are following the SDMX statistical metadata codelists, and resemble the SMDX concepts used by international statistical agencies. (See more technical information here.)

Small organizations often cannot afford to have data engineers and data scientists on staff, and they employ analysts who work with Excel, OpenOffice, PowerBI, SPSS or Stata. The problem with these applications is that they often require the user to manually adjust the data, with keyboard entries or mouse clicks. Furthermore, they do not provide a precise logging of the data processing, manipulation history. The manual data processing and manipulation is very error prone and makes the use of complex and high value resources, such as harmonized surveys or symmetric input-output tables, to name two important source we deal with, impossible to use. The use of these high-value data sources often requires tens of thousands of data processing steps: no human can do it faultlessly.

What is even more problematic that simple applications for analysis do not provide a log of these manipulations’ steps: pulling over a column with the mouse, renaming a row, adding a zero to an empty cell. This makes senior supervisory oversight and external audit very costly.

Our data comes with full history: all changes are visible, and we even open the code or algorithm that processed the raw data. Your analysts can still use their favourite spreadsheet or statistical software application, but they can start from a clean, tidy dataset, with all data wrangling, currency and unit conversion, imputation and other low-priority but important tasks done and logged.

Survey Harmonization

Mon, 05 Jul 2021 08:00:00 +0000

We provide retrospecitve, ex post, and ex ante survey harmonization to our partners.

The aim of retrospective survey harmonization is to pool data from pre-existing surveys made with a similar methodology in different points in time and different countries or territories. Ex post survey harmonization is in a way a passive form of pooling research funding because you can utilize information from surveying that were made on somebody else’s expense.

The Arab Barometer surveys do not have a consolidated codebook, but our retroharmonize software created one, and put together data from three years and collected in many countries about various public policy issues.

The aim of ex ante survey harmonization is to maximize the value from future retrospective harmonization; in a way, it is an active form of pooling research funding, because you benefit from money spent on related open governmental and open science survey programs.

In this example we designed a survey representative among music professionals that it can be compared with large-sample, national surveys on living conditions and attitudes, and with occupational groups. Nationally representative surveys do not question enough musicians to allow such specific use; musician only surveys do not allow comparison.

retorhamonize is a peer-reviewed, scientfic statistcal software that allows the programmatic retrospective harmonization of surveys, such as the last 35 years of all Eurobarometer microdata, or all Afrobarometer microdata. Eurobarometer grew out of certain CEE member states’ need for comparable data about their music and audiovisual sectors. We commissioned surveys following ESSNet-Culture guidelines and combined our survey data with open access European microdata-level surveys.

regions solves the problems caused by Europe’s shifting regional boundaries, which have undergone changes in several thousand places over the last twenty years, meaning member states’ and Eurostat’s regional statistics are not comparable over more than two to three years. This software validates and, where possible, changes the regional coding from NUTS1999 until the not yet used NUTS2021, opening up vast, valuable, untapped data sources that can be used for longitudinal analysis or for panel analysis far more precise than what national data alone would allow. It was originally designed in a research project at IVIR in the University of Amsterdam to understand the geographical dynamics of book piracy. Because of the needs this software fills, it had 700 users in the first month after publication. It is particularly useful to re-code old surveys, as regional boundaries are changing in each decade several hundred times in Europe.

Including Indicators from Arab Barometer in Our Observatory

Mon, 28 Jun 2021 09:00:00 +0200

A new version of the retroharmonize R package – which is working with retrospective, ex post harmonization of survey data – was released yesterday after peer-review on CRAN. It allows us to compare opinion polling data from the Arab Barometer with the Eurobarometer and Afrorbarometer. This is the first version that is released in the rOpenGov community, a community of R package developers on open government data analytics and related topics.

Surveys are the most important data sources in social and economic statistics – they ask people about their lives, their attitudes and self-reported actions, or record data from companies and NGOs. Survey harmonization makes survey data comparable across time and countries. It is very important, because often we do not know without comparison if an indicator value is low or high. If 40% of the people think that climate change is a very serious problem, it does not really tell us much without knowing what percentage of the people answered this question similarly a year ago, or in other parts of the world.

With the help of Ahmed Shabani and Yousef Ibrahim, we created a third case study after the Eurobarometer, and Afrobarometer, about working with the Arab Barometer harmonized survey data files.

Ex ante survey harmonization means that researchers design questionnaires that are asking the same questions with the same survey methodology in repeated, distinct times (waves), or across different countries with carefully harmonized question translations. Ex post harmonizations means that the resulting data has the same variable names, same variable coding, and can be joined into a tidy data frame for joint statistical analysis. While seemingly a simple task, it involves plenty of metadata adjustments, because established survey programs like Eurobarometer, Afrobarometer or Arab Barometer have several decades of history, and several decades of coding practices and file formatting legacy.

Variable harmonization means that if the same question is called in one microdata source Q108 and the other eval-parl-elections then we make sure that they get a harmonize and machine readable name without spaces and special characters.
Variable label harmonization means that the same questionnaire items get the same numeric coding and same categorical labels.
Missing case harmonization means that various forms of missingness are treated the same way.

For the climate awareness dataset get the country averages and aggregates from Zenodo, and the plot in jpg or png from figshare.

In our new Arab Barometer case study, the evaulation of parliamentary elections has the following labels. We code them consistently 1 = free_and_fair, 2 = some_minor_problems, 3 = some_major_problems and 4 = not_free.

“0. missing”	“1. they were completely free and fair”
“2. they were free and fair, with some minor problems”	“3. they were free and fair, with some major problems”
“4. they were not free and fair”	“8. i don’t know”
“9. declined to answer”	“Missing”
“They were completely free and fair”	“They were free and fair, with some minor breaches”
“They were free and fair, with some major breaches”	“They were not free and fair”
“Don’t know”	“Refuse”
“Completely free and fair”	“Free and fair, but with minor problems”
“Free and fair, with major problems”	“Not free or fair”
“Don’t know (Do not read)”	“Decline to answer (Do not read)”

Of course, this harmonization is essential to get clean results like this:

For evaluation or reuse of parliamentary elections dataset get the replication data and the code from the Zenodo open repository.

In our case study, we had three forms of missingness: the respondent did not know the answer, the respondent did not want to answer, and at last, in some cases the respondent was not asked, because the country held no parliamentary elections. While in numerical processing, all these answers must be left out from calculating averages, for example, in a more detailed, categorical analysis they represent very different cases. A high level of refusal to answer may be an indicator of surpressing democratic opinion forming in itself.

Survey harmonization with many countries entails tens of thousands of small data management task, which, unless automatically documented, logged, and created with a reproducible code, is a helplessly error-prone process. We believe that our open-source software will bring many new statistical information to the light, which, while legally open, was never processed due to the large investment needed.

We also started building experimental APIs data is running retroharmonize regularly. We will place cultural access and participation data in the Digital Music Observatory, climate awareness, policy support and self-reported mitigation strategies into the Green Deal Data Observatory, and economy and well-being data into our Economy Data Observatory.

Further plans

Retrospective survey harmonization is a far more complex task than this blogpost suggest. Retrospective survey harmonization is a far more complex task than this blogpost suggest, because established survey programs have gathered decades of legacy data in legacy coding schemes and legacy file formats. Putting the data right, and especially putting the invaluable descriptive and administrative (processing) metadata right is a huge undertaking. We are releasing example codes, datasets and charts for researchers to comapre our harmonized results with theirs, and improve our software. We are releasing example codes, datasets and charts for researchers to comapre our harmonized results with theirs, and improve our software.

Use our software

The retroharmonize R package can be freely used, modified and distributed under the GPL-3 license. For the main developer and contributors, see the package homepage. If you use it for your work, please kindly cite it as:

Daniel Antal (2021). retroharmonize: Ex Post Survey Data Harmonization. R package version 0.1.17. https://doi.org/10.5281/zenodo.5034752

Download the BibLaTeX entry.

Tutorial to work with the Arab Barometer survey data

Daniel Antal, & Ahmed Shaibani. (2021, June 26). Case Study: Working With Arab Barometer Surveys for the retroharmonize R package (Version 0.1.6). Zenodo. https://doi.org/10.5281/zenodo.5034759

For the replication data to report potential issues and improvement suggestions with the code:

Daniel Antal, & Ahmed Shaibani. (2021). Replication Data for the retroharmonize R Package Case Study: Working With Arab Barometer Surveys (Version 0.1.6) [Data set]. Zenodo. https://doi.org/10.5281/zenodo.5034741

Experimental API

We are also experimenting with the automated placement of authoritative and citeable figures and datasets in open repositories. For the climate awareness dataset get the country averages and aggregates from Zenodo, and the plot in jpg or png from figshare. Our plan is to release open data in a modern API with rich descriptive metadata meeting the Dublin Core and DataCite standards, and further administrative metadata for correct coding, joining and further manipulating or data, or for easy import into your database.

Join our open source effort

Want to help us improve our open data service? Include Lationbarómetro and the Caucasus Barometer in our offering? Join the rOpenGov community of R package developers, an our open collaboration to create the automated data observatories. We are not only looking for developers, but data curators and service design associates, too.

Competition Data Observatory

Wed, 23 Jun 2021 00:00:00 +0000

*Our observatory is monitoring the certain segments of the European economy, and develops tools for computational antitrust in Europe. We take a critical SME-, intellectual property policy and competition policy point of view automation, robotization, and the AI revolution on the service-oriented European social market economy.

We aim to create early-warning, risk, economic effect, and impact indicators that can be used in scientific, business and policy contexts for professionals who are working on re-setting the European economy after a devastating pandemic and in the age of AI. We would like to map data between economic activities (NACE), antitrust markets, and sub-national, regional, metropolitian area data.

Visit the Competition Data Observatory

Try the Competition Data Observatory API

Connect on LinkedIn

Finding reliable historic and new data that can fuel large market monitoring schemes or computational antitrust models is surprisingly hard. And it is even more hopeless if you work as a (data) journalist, a policy researcher in an NGO, or in a competition law practice that does not provide you with an army of (geo)statisticians, data engineers, and data scientists who can render various data into usable format, i.e. something that you can trust, quote, visualize, import, or copy & paste.

Better, Bigger, Faster, More

Novel data products: Official statistics at the national and European levels follow legal regulations, and in the EU, compromises between member states. New policy indicators often appear 5-10 years after demand appears. We employ the same methodology, software, and often even the same data that Eurostat might use to develop policy indicators, but we do not have to wait for a political and legal consensus to create new datasets. See our example 100,000 Opinions on the Most Pressing Global Problem.

Better data: Statistical agencies, old fashioned observatories, and data providers often do not have the mandate, know-how or resources to improve data quality. Using peer-reviewed statistical software and hundreds of computational tests, we are able to correct mistakes, impute missing data, generate forecasts, and increase the information content of public data by 20-200% percent. This makes the data usable for NGOs, journalists, and visual artists—among other potential users—who do not have this statistical know-how to make incomplete, mislabelled or low quality data usable for their needs and applications. See our example with the indicator Turnover of the Radio Broadcasting Industry in Europe

Never seen data: The 2019/1024 directive on open data and the re-use of public sector information of the European Union (which is an extension and modernization of the earlier directives on re-use of public sector information since 2003) makes data gathered in EU institutions, national institutions, and municipalities, as well as state-owned companies legally available. According to the European Data Portal the estimated historical cost of the data released annually is in the billions of euros. But if this data is a gold mine, its full potential can only be unlocked by an experienced data mining partner like Reprex. Here is why: data is not readily downloadable; it sits in various obsolete file formats in disorganized databases; it is documented in various languages, or not documented at all; it is plagued with various processing errors. We make the powerful promise](http://dataobservatory.eu/post/2021-06-18-gold-without-rush/) of the EU legislation a reality in the field of the Green Deal policy context.

Increase Your Impact, Avoid Old Mistakes

Impact: We publish the data in a way that it is easy to find—as a separate data publication with a DOI, full library metadata, and place it in open science repositories. Our data is more findable than 99% of the open science data, and therefore makes far bigger impact. See our data on the European open science repository Zenodo managed by CERN (the European Organization for Nuclear Research).

Easy-to-use data: Our data follows the tidy data principle and comes with all the recommended Dublin Core and DataCite metadata. This increases our data compatibility, allowing users to open it in any spreadsheet application or import into their databases. We publish the data in tabular form, and in JSON form through our API enabling automatic retrieval for heavy users, especially if they plan to automatically use our data in daily or weekly updates. Using the best practice of data formatting and documentation with metadata ensures reproducibility and data integrity, rather than repeating data processing and preparation steps (e.g. changing data formats, removing unwanted characters, creating documentation, and other data processing steps that take up thousands of working hours. See our blogpost on the data Sisyphus.

Ethical Big Data for All

Big picture: Integrating and joining data is hard—it requires engineering, mathematical, and geo-statistical know-how that a large amount of environmental users and stakeholders do not possess. Some examples of the challenges implicit in making data usable include addressing the changing boundaries of French departments (and European administrative-geographic borders, in general), various projections of coordinates on satellite images of land cover, different measurement areas for public opinion and hydrological data, public finance expressed in different orders (e.g. millions versus thousands of euros). We create data that is easy to combine, map, and visualize for end users. See our case study on the severity and awareness of flood risk in Belgium, as well as the financial capacity to manage it.

Open Collaboration

Find us on LinkedIn or send us an email.

Open Data - The New Gold Without the Rush

Fri, 18 Jun 2021 17:00:00 +0200

If open data is the new gold, why even those who release fail to reuse it? We created an open collaboration of data curators and open-source developers to dig into novel open data sources and/or increase the usability of existing ones. We transform reproducible research software into research- as-service.

There is no rush for it, because panning out its value requires a lot of hours of hard work. Our goal is to automate this work to make open data usable at scale, even in trustworthy AI solutions.

Most open data is not public, it is not downloadable from the Internet – in the EU parlance, “open” only means a legal entitlement to get access to it. And even in the rare cases when data is open and public, often it is mired by data quality issues. We are working on the prototypes of a data-as-service and research-as-service built with open-source statistical software that taps into various and often neglected open data sources.

We are in the prototype phase in June and our intentions are to have a well-functioning service by the time of the conference, because we are working only with open-source software elements; our technological readiness level is already very high. The novelty of our process is that we are trying to further develop and integrate a few open-source technology items into technologically and financially sustainable data-as-service and even research-as-service solutions.

Our review of about 80 EU, UN and OECD data observatories reveals that most of them do not use these organizations’s open data - instead they use various, and often not well processed proprietary sources.

We are taking a new and modern approach to the data observatory concept, and modernizing it with the application of 21st century data and metadata standards, the new results of reproducible research and data science. Various UN and OECD bodies, and particularly the European Union support or maintain more than 60 data observatories, or permanent data collection and dissemination points, but even these do not use these organizations and their members open data. We are building open-source data observatories, which run open-source statistical software that automatically processes and documents reusable public sector data (from public transport, meteorology, tax offices, taxpayer funded satellite systems, etc.) and reusable scientific data (from EU taxpayer funded research) into new, high quality statistical indicators.

We are taking a new and modern approach to the ‘data observatory’ concept, and modernizing it with the application of 21st century data and metadata standards, the new results of reproducible research and data science

We are building various open-source data collection tools in R and Python to bring up data from big data APIs and legally open, but not public, and not well served data sources. For example, we are working on capturing representative data from the Spotify API or creating harmonized datasets from the Eurobarometer and Afrobarometer survey programs.
Open data is usually not public; whatever is legally accessible is usually not ready to use for commercial or scientific purposes. In Europe, almost all taxpayer funded data is legally open for reuse, but it is usually stored in heterogeneous formats, processed into an original government or scientific need, and with various and low documentation standards. Our expert data curators are looking for new data sources that should be (re-) processed and re-documented to be usable for a wider community. We would like to introduce our service flow, which touches upon many important aspects of data scientist, data engineer and data curatorial work.
We believe that even such generally trusted data sources as Eurostat often need to be reprocessed, because various legal and political constraints do not allow the common European statistical services to provide optimal quality data – for example, on the regional and city levels.
With rOpenGov and other partners, we are creating open-source statistical software in R to re-process these heterogenous and low-quality data into tidy statistical indicators to automatically validate and document it.
We are carefully documenting and releasing administrative, processing, and descriptive metadata, following international metadata standards, to make our data easy to find and easy to use for data analysts.
We are automatically creating depositions and authoritative copies marked with an individual digital object identifier (DOI) to maintain data integrity.
We are building simple databases and supporting APIs that release the data without restrictions, in a tidy format that is easy to join with other data, or easy to join into databases, together with standardized metadata.
We maintain observatory websites (see: Digital Music Observatory, Green Deal Data Observatory, Economy Data Observatory) where not only the data is available, but we provide tutorials and use cases to make it easier to use them. Our mission is to show a modern, 21st century reimagination of the data observatory concept developed and supported by the UN, EU and OECD, and we want to show that modern reproducible research and open data could make the existing 60 data observatories and the planned new ones grow faster into data ecosystems.

We are working around the open collaboration concept, which is well-known in open source software development and reproducible science, but we try to make this agile project management methodology more inclusive, and include data curators, and various institutional partners into this approach. Based around our early-stage startup, Reprex, and the open-source developer community rOpenGov, we are working together with other developers, data scientists, and domain specific data experts in climate change and mitigation, antitrust and innovation policies, and various aspects of the music and film industry.

Our open collaboration is truly open: new data curators,developers and service designers, even volunteers and citizen scientists are welcome to join.

Our open collaboration is truly open: new data curators, data scientists and data engineers are welcome to join. We develop open-source software in an agile way, so you can join in with an intermediate programming skill to build unit tests or add new functionality, and if you are a beginner, you can start with documentation and testing our tutorials. For business, policy, and scientific data analysts, we provide unexploited, exciting new datasets. Advanced developers can join our development team: the statistical data creation is mainly made in the R language, and the service infrastructure in Python and Go components.

Music Creators’ Earnings in the Streaming Era

Fri, 18 Jun 2021 08:00:00 +0200

The idea of our Digital Music Observatory was brought to the UK policy debate on music streaming by the Written evidence submitted by The state51 Music Group to the Economics of music streaming review of the UK Parliaments' DCMS Committee¹.

The music industry requires a permanent market monitoring facility to win fights in competition tribunals, because it is increasingly disputing revenues with the world’s biggest data owners. This was precisely the role of the former CEEMID² program, which was initiated by a group of collective management societies. Starting with three relatively data-poor countries, where data pooling allowed rightsholders to increase revenues, the CEEMID data collection program was extended in 2019 to 12 countries.The final regional report, after the release of the detailed Hungarian, Slovak and Croatian reports of CEEMID was sponsored by Consolidated Independent (of the state51 music group.)

CEEMID was eventually to formed into the Demo Music Observatory in 2020³, following the planned structure of the European Music Observatory, and validated in the world’s 2nd ranked university-backed incubator, the Yes!Delft AI+Blockchain Validation Lab. In 2021, under the final name Digital Music Observatory, it became open for any rightsholder or stakeholder organization or music research institute, and it is being launched with the help of the JUMP European Music Market Accelerator Programme which is co-funded by the Creative Europe Programme of the European Union.

In December 2020, we started investigating how the music observatory concept could be introduced in the UK, and how our data and analytical skills could be used in the Music Creators’ Earnings in the Streaming Era (in short: MCE) project, which is taking place paralell to the heated political debates around the DCMS inquiry. After the state51 music group gave permission for the UK Intellectual Property Office to reuse the data that was originally published as the experimental CEEMID-CI Streaming Volume and Revenue Indexes, we came to a cooperation agreement between the MCE Project and the Digital Music Observatory. We provided a detailed historical analysis and computer simulation for the MCE Project, and we will host all the data of the Music Creators’ Earnings Report in our observatory, hopefully no later than early July 2021.

The Digital Music Observatory contributes to the Music Creators’ Earnings in the Streaming Era project with understanding the level of justified and unjustified differences in rightsholder earnings, and putting them into a broader music economy context.

We started our cooperation with the two principal investigators of the project, Prof David Hesmondhalgh and Dr Hyojugn Sun back in April and will start releasing the findings and the data in July 2021.

Justified and Unjustified Differences in Earnings

Stating that the greatest difference among rightsholders’ earnings is related to the popularity of their works and recorded fixations can appear banal and trivial. Yet, because many payout problems appear in the hard-to-describe long tail, understanding the justified differences of rightsholder earnings is an important step towards identifying the unjustified differences. It would be a breach of copyright law if less popular, or never played artists, would receive significantly more payment at the expense of popular artists from streaming providers. The earnings must reflect the difference in use and the economic value in use among rightsholders.

In our analysis we quantify differences using the actual data of the CEEMID-CI Streaming Indexes, created from hundreds of millions of data points, and computer simulations under realistic scenarios.

Justified Difference & Changes Over Time

Among the justified differences we quantify four objective justifications:

The variability of the domestic price of a stream over time shows a diminishing, but variable value of streams. Depending on the release date of a recording, and how quickly it builds up or loses the interest of the audience, the same number of streams can result in about 28% different earnings. In the period 2015-2019, later releases were facing diminishing revenues on streaming platforms.
The variability of international market share and international streaming prices in International Competitiveness. Compared to UK streaming prices, most international markets, particularly emerging markets, have a much greater variability of streaming prices. The variability of prices in advanced foreign markets such as Germany was similar to that of the British market, but in emerging markets and smaller advanced markets–such as the Netherlands–we measured a variability of around 50-80%. Artists who have a significant foreign presence, depending on their foreign market share, can experience 2-3 times greater differences in earnings than artists whose audience is predominantly British.

In our simulated results, the depreciation of the GBP shielded the internationally competitive rightsholders from a significant part of the otherwise negative price change in streaming markets.

The variability of the exchange rate that is applied when translating foreign currency revenues to the British pound in Exchange Rate Effects. Our CEEMID-CI Streaming Indexes index covers the post-Brexit referendum period, when the British pound was generally depreciating against most currencies. This resulted in a GBP-denominated translation gain for artists with a foreign presence. We show that the variability of the GBP exchange rates can add bigger justified differences among rightsholders’ earnings than the entire British price variation. The exchange rate movements are typically in the range of 30%, or at the level of the British domestic price variations in streaming prices. In our simulated results, this effect shielded the internationally competitive rightsholders from a significant part of the otherwise negative price change in foreign markets.
We were also investigating the choice of distribution model. , i.e., Both models, the currently used pro-rata model and the user-centric distribution, which has many proponents (and was introduced by SoundCloud in 2021), changes the earnings of artists. We think that both models represent a bad compromise, but they are legal, and a change of zero-sum distribution change could potentially increase the income of less popular and older artists at the expense of very popular and younger artists. More about this in our forthcoming report!

In our understanding, there are some known and some hypothetical causes of unjustified earning differences.

We did not have systemic data on the uncollected revenues–these are earnings that are legally made, but due to documentation, matching, processing, accounting, or other problems, the earnings are not paid. We could not even attempt to estimate this problem in the absence of relevant British empirical data. The problem is likely to be greater in the case of composers than in producer and performer revenues. In ideal cases, of course, the unclaimed royalty is near 0% of the earnings; it seems that on advanced markets the magnitude of this problem is in the single digits, and in emerging markets much greater, sometimes up to 50%. Royalty distribution is a costly business, and the smaller the revenue, the smaller the cost base to manage billions of transactions and related micropayments. We are working with a large group of eminent copyright researchers to understand this program better and provide regulatory solutions. See Recommendation Systems: What can Go Wrong with the Algorithm? - Effects on equitable remuneration, fair value, cultural and media policy goals
More analysis is required to understand how the algorithmic, highly autonomous recommendation systems of digital platforms such as Spotify, YouTube, Apple, and Deezer, among others, impact music rightsholders’ earnings. There are some empirical findings that suggest that such biases are present in various platforms, but due to the high complexity of recommendation systems, it is impossible to intuitively assign blame to pre-existing user biases, wrong training datasets, improper algorithm design, and other factors. We are working with our data curators, competition economist Dr Peter Ormosi, antropologist and data scientist Dr Botond Vitos and musicologist Dominika Semaňáková - We Want Machine Learning Algorithms to Learn More About Slovak Music to understand what can go wrong here (see Trustworthy AI: Check Where the Machine Learning Algorithm is Learning From).
There is always a hypothetical possibility that organizations with monopolistic power try to corner the market or make the playing field uneven. The music industry requires a permanent market monitoring facility to win fights in competition tribunals, because it is increasingly disputing revenues with the world’s biggest data owners. We are working with our data curators, competition economist Dr Peter Ormosi and copyright lawyer dr Eszter Kabai - New Indicators for Royalty Pricing and Music Antitrust to find potential traces of an uneven playing field (see: Music Streaming: Is It a Level Playing Field?.)

Solidarity & Equitable Remuneration

Equitable remuneration is a legal concept which has an economic aspect. In international law, it simply means that men and women should receive equal pay for equal work. Within the context of international copyright law, it was introduced by the Rome Convention, and it means that equitable remuneration means the same payment for the same use, regardless of genre, gender or other unrelated characteristics of the rightsholder.

While the word equitable in everyday usage often implies some level of equality and solidarity, in the context of royalty payments, these terms should not be mixed.

Digital Music Observatory uses harmonized, anonimous surveys conducted among musicians to find out about their living conditions compared to their peers in other countries, and people in other professions.

Solidarity is present in many royalty payout schemes, but it is unrelated to the legal concept of equitable remuneration. Music earnings are very heavily skewed towards a small number of very successful composers, performers, and producers. The music streaming licensing model has little elements of solidarity, unlike some of the licensing models that it is replacing–particularly public performance licensing. However, in those cases, the solidarity element is decided by the rightsholders themselves, and not by external parties like radio broadcasters or streaming service providers.

The so-called socio-cultural funds that provide assistance for artists in financial need must be managed by rightsholders, not others, and the current streaming model makes the organizations of such solidaristic action particularly difficult. Our data curator, Katie Long is working on us to find metrics and measurement possibilities on solidarity among rightsolders.

The Size of the Pie and the Distribution of the Pie

The current debate in the United Kingdom is often organized around the submission of the #BrokenRecord campaign to the DCMS committee, which calls for a legal re-definition of equitable remuneration rights⁴. This idea is not unique to the United Kingdom, in various European jurisdictions, performers fought similar campaigns for legislation or went to court, sometimes successfully, for example, in Hungary.

Another hot redistribution topic is the choice between the so-called pro-rata versus user-centric distribution of streaming royalties. We think that both models represent a bad compromise, but they are legal, and a zero-sum distribution change could potentially increase the income of less popular and older artists at the expense of very popular and younger artists. We instead propose an alternative approach of artist-centric distribution that could be potentially a win-win for all rightsholder groups; further elaboration of the concept lies outside the scope of this report.

Digital Music Observatory uses monitors volumes, prices and revenues on the total market, and compares them to calculated fair values. By understand the entire music economy, we can highlight when music is devaluing in various uses or countries.

Our mission with the Digital Music Observatory is to help focus the policy debate with facts around the economics of music streaming. The legal concept equitable remuneration is inseparable from the economic concept of fair valuation, and music streaming earnings cannot be subject to a valid economic analysis without analyzing the economics of music. Streaming services are competing with digital downloads, physical sales, and radio broadcasting; and the media streaming of YouTube and similar services is competing with music streaming, radio, and television broadcasting as well as retransmissions. It is a critically important to determine if in replacing earlier services and sales channels, the new streaming licensing model (a mix of the mechanical and public performance licensing) is also capable of replacing the revenues for all rightsholders.

The current streaming licensing model in Europe is a mix of mechanical and public performance rights. Therefore, when we are talking about music streaming, we must compare the streaming sub-market with the digital downloads, physical sales and private copying markets (mechanical licensing), and with the radio, television, cable and satellite retransmission markets (public performance licensing.) Our experience outside the UK suggests that these replacement values are very low.

Join us

Join our open collaboration Music Data Observatory team as a data curator, developer or business developer. More interested in antitrust, innovation policy or economic impact analysis? Try our Economy Data Observatory team! Or your interest lies more in climate change, mitigation or climate action? Check out our Green Deal Data Observatory team!

Footnote References

state51 Music Group. 2020. “Written Evidence Submitted by The state51 Music Group. Economics of Music Streaming Review. Response to Call for Evidence.” UK Parliament website. https://committees.parliament.uk/writtenevidence/15422/html/. ↩︎
Artisjus, HDS, SOZA, and Candole Partners. 2014. “Measuring and Reporting Regional Economic Value Added, National Income and Employment by the Music Industry in a Creative Industries Perspective. Memorandum of Understanding to Create a Regional Music Database to Support Professional National Reporting, Economic Valuation and a Regional Music Study.” ↩︎
Antal, Daniel. 2021. “Launching Our Demo Music Observatory.” Data & Lyrics. Reprex. https://dataandlyrics.com/post/2020-09-15-music-observatory-launch/. ↩︎
Gray, Tom. 2020. “#BrokenRecord Campaign Submission (Supplementary to Oral Evidence).” UK Parliament website. https://committees.parliament.uk/writtenevidence/15512/html/. ↩︎

Analyze Locally, Act Globally: New regions R Package Release

Wed, 16 Jun 2021 12:00:00 +0200

The new version of our rOpenGov R package regions was released today on CRAN. This package is one of the engines of our experimental open data-as-service Green Deal Data Observatory, Economy Data Observatory, Digital Music Observatory prototypes, which aim to place open data packages into open-source applications.

In international comparison the use of nationally aggregated indicators often have many disadvantages: they inhibit very different levels of homogeneity, and data is often very limited in number of observations for a cross-sectional analysis. When comparing European countries, a few missing cases can limit the cross-section of countries to around 20 cases which disallows the use of many analytical methods. Working with sub-national statistics has many advantages: the similarity of the aggregation level and high number of observations can allow more precise control of model parameters and errors, and the number of observations grows from 20 to 200-300.

The change from national to sub-national level comes with a huge data processing price: internal administrative boundaries, their names, codes codes change very frequently.

Yet the change from national to sub-national level comes with a huge data processing price. While national boundaries are relatively stable, with only a handful of changes in each recent decade. The change of national boundaries requires a more-or-less global consensus. But states are free to change their internal administrative boundaries, and they do it with large frequency. This means that the names, identification codes and boundary definitions of sub-national regions change very frequently. Joining data from different sources and different years can be very difficult.

Our regions R package helps the data processing, validation and imputation of sub-national, regional datasets and their coding.

There are numerous advantages of switching from a national level of the analysis to a sub-national level comes with a huge price in data processing, validation and imputation, and the regions package aims to help this process.

You can review the problem, and the code that created the two map comparisons, in the Maping Regional Data, Maping Metadata Problems vignette article of the package. A more detailed problem description can be found in Working With Regional, Sub-National Statistical Products.

This package is an offspring of the eurostat package on rOpenGov. It started as a tool to validate and re-code regional Eurostat statistics, but it aims to be a general solution for all sub-national statistics. It will be developed parallel with other rOpenGov packages.

Get the Package

You can install the development version from GitHub with:

devtools::install_github("rOpenGov/regions")

or the released version from CRAN:

install.packages("regions")

You can review the complete package documentation on regions.dataobservaotry.eu. If you find any problems with the code, please raise an issue on Github. Pull requests are welcome if you agree with the Contributor Code of Conduct

If you use regions in your work, please cite the package as: Daniel Antal. (2021, June 16). regions (Version 0.1.7). CRAN. http://doi.org/10.5281/zenodo.4965909

Download the BibLaTeX entry.

Join us

Join our open collaboration Green Deal Data Observatory team as a data curator, developer or business developer. More interested in antitrust, innovation policy or economic impact analysis? Try our Economy Data Observatory team! Or your interest lies more in data governance, trustworthy AI and other digital market problems? Check out our Digital Music Observatory team!

Open Data is Like Gold in the Mud Below the Chilly Waves of Mountain Rivers

Thu, 10 Jun 2021 07:00:00 +0200

Open data is like gold in the mud below the chilly waves of mountain rivers. Panning it out requires a lot of patience, or a good machine.

As the founder of the automated data observatories that are part of Reprex’s core activities, what type of data do you usually use in your day-to-day work?

The automated data observatories are results of syndicated research, data pooling, and other creative solutions to the problem of missing or hard-to-find data. The music industry is a very fragmented industry, where market research budgets and data are scattered in tens of thousands of small organizations in Europe. Working for the music and film industry as a data analyst and economist was always a pain because most of the efforts went into trying to find any data that can be analyzed. I spent most of the last 7-8 years trying to find any sort of information—from satellites to government archives—that could be formed into actionable data. I see three big sources of information: textual,numeric, and continuous recordings for on-site, offsite, and satellite sensors. I am much better with numbers than with natural language processing, and I am improving with sensory sources. But technically, I can mint any systematic information—the text of an old book, a satellite image, or an opinion poll—into datasets.

For you, what would be the ultimate dataset, or datasets that you would like to see in the Green Deal Data Observatory?

Our retroharmonize and regions packages can create regional statistics from Eurobarometer and Afrobarometer surveys on how people think locally about climate change. I would like to combine this with local information on observable climate change, such as drought, urban heat, and extreme weather conditions. Do people have to feel the pain of climate change to believe in the phenomenon? How do self-reported mitigation steps correlate with what people already feel in their local environment? Suzan is talking about measuring mitigation and damage control, because she’s aware of the already present health risks in overheating urban environments. I am more interested in what people think.

See our case study on connecting local tax revenues, climate awareness poll data and drought data in Belgium - we want to extend this to Europe and then to Africa. We also published the code how to do it with tutorials 1, 2 for our International Open Data Day 2021 Event.

Is there a number or piece of information that recently surprised you? If so, what was it?

There were a few numbers that surprised me, and some of them were brought up by our observatory teams. Karel is talking about the fact that not all green energy is green at all: many hydropower stations contribute to the greenhouse effect and not reduce it. Annette brought up the growing interest in the Dalmatian breed after the Disney 101 Dalmatians movies, and it reminded me of the astonishing growth in interest for chess sets, chess tutorials, and platform subscriptions after the success of Netflix’s The Queen’s Gambit.

The Queen’s Gambit’ Chess Boom Moves Online By Rachael Dottle on bloomberg.com

Annette is talking about the importance of cultural influencers, and on that theme, what could be more exciting that Netflix’s biggest success so far is not a detective series or a soap opera but a coming-of-age story of a female chess prodigy. Intelligence is sexy, and we are in the intelligence business.

But to tell a more serious and more sobering number, I recently read with surprise that there are more people smoking cigarettes on Earth in 2021 than in 1990. Population growth in developing countries replaced the shrinking number of developed country smokers. While I live in Europe, where smoking is strongly declining, it reminds me that Europe’s population is a small part of the world. We cannot take for granted that our home-grown experiences about the world are globally valid.

Do you have a good example of really good, or really bad use of data?

FiveThirtyEight.com had a wonderful podcast series, produced by Jody Avirgan, called What’s the Point. It is exactly about good and bad uses of data, and each episode is super interesting. Maybe the most memorable is Why the Bronx Really Burned. New York City tried to measure fire response times, identify redundancies in service, and close or re-allocate fire stations accordingly. What resulted, though, was a perfect storm of bad data: The methodology was flawed, the analysis was rife with biases, and the results were interpreted in a way that stacked the deck against poorer neighborhoods. It is similar to many stories told in a very compelling argument by Catherine D’Ignazio and Lauren F. Klein in their much celebrated book, Data Feminism. Usually, the bad use of data starts with a bad data collection practice. Data analysts in corporations, NGOs, public policy organizations and even in science usually analyze the data that is available.

You can find these examples, together with many more that our contributors recommend, in the motivating examples of Create New Datasets and the Remain Critical parts of our onboarding material. We hope that more and more professionals and citizen scientist will help us to create high-quality and open data.

The real power lies in designing a data collection program. A consistent data collection program usually requires an investment that only powerful organizations, such as government agencies, very large corporations, or the richest universities can afford. You cannot really analyze the data that is not collected and recorded; and usually what is not recorded is more interesting than what is. Our observatories want to democratize the data collection process and make it more available, more shared with research automation and pooling.

You cannot really analyze the data that is not collected and recorded; and usually what is not recorded is more interesting than what is. Our observatories want to democratize the data collection process and make it more available, more shared with research automation and pooling.

From your perspective, what do you see being the greatest problem with open data in 2021?

I have been involved with open data policies since 2004. The problem has not changed much: more and more data are available from governmental and scientific sources, but in a form that makes them useless. Data without clear description and clear processing information is useless for analytical purposes: it cannot be integrated with other data, and it cannot be trusted and verified. If researchers or government entities that fall under the Open Data Directive release data for reuse in a way that does not have descriptive or processing metadata, it is almost as if they did not release anything. You need this additional information to make valid analyses of the data, and to reverse-engineer them may cost more than to recollect the data in a properly documented process. Our developers, particularly Leo and Pyry are talking eloquently about why you have to be careful even with governmental statistical products, and constantly be on the watch out for data quality.

Our API is not only publishing descriptive and processing metadata alongside with our data, but we also make all critical elements of our processing code available for peer-review on rOpenGov

What do you think the Green Deal Data Observatory, and our other automated observatories do, to make open data more credible in the European economic policy community and be accepted as verified information?

Most of our work is in research automation, and a very large part of our efforts are aiming to reverse engineer missing descriptive and processing metadata. In a way, I like to compare ourselves to the working method of the open-source intelligence platform Bellingcat. They were able to use publicly available, scattered information from satellites and social media to identify each member of the Russian military company that illegally entered the territory of Ukraine and shot down the Malaysian Airways MH17 with 297, mainly Dutch, civilians on board.

How we create value for research-oriented consultancies, public policy institutes, university research teams, journalists or NGOs.

We do not do such investigations but work very similarly to them in how we are filtering through many data sources and attempting to verify them when their descriptions and processing history is unknown. In the last years, we were able to estore the metadata of many European and African open data surveys, economic impact, and environmental impact data, or many other open data that was lying around for many years without users.

Open data is like gold in the mud below the chilly waves of mountain rivers. Panning it out requires a lot of patience, or a good machine. I think we will come to as surprising and strong findings as Bellingcat, but we are not focusing on individual events and stories, but on social and environmental processes and changes.

Join us

Trustworthy AI: Check Where the Machine Learning Algorithm is Learning From

Tue, 08 Jun 2021 12:10:00 +0200

We do care what our children learn, but we do not care yet about what our robots learn from. One key idea behind trustworthy AI is that you verify what data sources your machine learning algorithms can learn from. As we have emphasised in our forthcoming academic paper and in our experiments, one key problem that goes wrong when you see too few small country artists, or too few womxn in the charts is that the big tech recommendation systems and other autonomous systems are learning from historically biased or patchy data.

This is precisely the type of work we are doing with the continued support of the Slovak national rightsholder organizations. In our work in Slovakia, we reverse engineered some of these undesirable outcomes. Our Slovak musicologist data curator, Dominika Semaňáková explains how we want to teach machine learning algorithms to learn more about Slovak music in her introductory interview.

A key mission of our Digital Music Observatory, which is our modern, subjective approach on how the future European Music Observatory should look like, is to not only to provide high-quality data on the music economy, the diversity of music, and the audience of music, but also on metadata. The quality and availability, interoperability of metadata (information about how the data should be used) is key to build trustworthy AI systems.

Traitors in a war used to be executed by firing squad, and it was a psychologically burdensome task for soldiers to have to shoot former comrades. When a 10-marksman squad fired 8 blank and 2 live ammunition, the traitor would be 100% dead, and the soldiers firing would walk away with a semblance of consolation in the fact they had an 80% chance of not having been the one that killed a former comrade. This is a textbook example of assigning responsibility and blame in systems. AI-driven systems such as the YouTube or Spotify recommendation systems, the shelf organization of Amazon books, or the workings of a stock photo agency come together through complex processes, and when they produce undesirable results, or, on the contrary, they improve life, it is difficult to assign blame or credit [..] If you do not see enough women on streaming charts, or if you think that the percentage of European films on your favorite streaming provider—or Slovak music on your music streaming service—is too low, you have to be able to distribute the blame in more precise terms than just saying “it’s the system” that is stacked up against women, small countries, or other groups. We need to be able to point the blame more precisely in order to effect change through economic incentives or legal constraints.

Assigning and avoding blame, read the earlier blogpost here.

This is precisely the type of work we are doing with the continued support of the Slovak national rightsholder organizations. In our work in Slovakia, we reverse engineered some of these undesirable outcomes. Popular video and music streaming recommendation systems have at least three major components based on machine learning. The problem is usually not that an algorithm is nasty and malicious; algorithms are often trained through “machine learning” techniques, and often, machines “learn” from biased, faulty, or low-quality information. Our Slovak musicologist data curator, Dominika Semaňáková explains how we want to teach machine learning algorithms to learn more about Slovak music in her introductory interview.

Read more about our Slovak music use case here.

These undesirable outcomes are sometimes illegal as they may go against non-discrimination or competition law. (See our ideas on what can go wrong – Music Streaming: Is It a Level Playing Field?) They may undermine national or EU-level cultural policy goals, media regulation, child protection rules, and fundamental rights protection against discrimination without basis. They may make Slovak artists earn significantly less than American artists.

In our academic (pre-print) paper we argue for new regulatory considerations to create a better, and more accountable playing field for deploying algorithms in a quasi-autonomous system, and we suggest further research to align economic incentives with the creation of higher quality and less biased metadata. The need for further research on how these large systems affect various fundamental rights, consumer or competition rights, or cultural and media policy goals cannot be overstated.

Incentives and investments into metadata

The first step is to open and understand these autonomous systems, and this is our mission with the Digital Music Observatory: it is a fully automated, open source, open data observatory that links public datasets in order to provide a comprehensive view of the European music industry. It produces key business and policy indicators, and research experiment data following the data pillars laid out in the Feasibility study for the establishment of a European Music Observatory.

Join our Digital Music Observatory as a user, curator, developer or help building our business case.

Join Copernicus Climate Data Store Data with Socio-Economic and Opinion Poll Data

Sun, 06 Jun 2021 10:00:00 +0200

In this series of blogposts we will show how to collect environmental data from the EU’s Copernicus Climate Data Store, and bring it to a data format that you can join with Eurostat’s socio-economic and environmental data. We have shown in a previous blogpost how to connect this to survey (opinion poll) and tax data, and a real policy problem in Belgium. We will create now subsequent tutorials to do more!

But first, why are we doing this? The European Union and its members states are releasing every year more and more data for open re-use since 2003, yet these are often not used in the EU’s data dissemination projects (the observatories) or in EU-funded research. We believe that there are many reasons behind this. Whilst more and more people can conduct business, scientific or policy analysis programmatically or with statistical software, knowledge how to systematically collect the data from the exponentially growing availability is not everybody’s specialty. And the lack of documentation, and high re-processing and validation need for open data is another drawback.

rOpenGov has long been producing high-quality, peer-reviewed R packages to work with open data, but their use is not for all. In an open collaboration, where you can join, too, rOpenGov teamed up with open source developers, knowledgeable data curators, and a service developer team lead by the Dutch reproducible research start-up Reprex to create a sustainable infrastructure that is permanently collecting, processing, documenting and visualizing open data. What we do is that we access open data (that is not always available for direct download) and re-process it to usable data that is tidy to be integrated with your existing data or databases. We are competing for the EU Datathon Challenge 1: supporting a European Green Deal agenda with open data as a service, and research as a servcie, and you are more than welcome to join our effort as a developer, a data curator, or as an occasional contributor to open government packages.

Register to the Copernicus Climate Data Store

Koen Hufkens, Reto Stauffer and Elio Campitelli created the ecmwfr R package for programmatically accessing the Copernicus Data Store service. Follow the CDS Functionality vignette to get started.

You will need to create a Register yourself for CDS services after accepting the Terms and conditions.

wf_set_key(user = "12345", 
           key = "00000000-aaaa-b1b1-0000-a1a1a1a1a1a1", 
           service = "cds")

You can check if you were successful with:

ecmwfr::wf_get_key(user = "12345", service = "cds")

Get the Data

Let us formulate our first request:

request_lai_hv_2019_06 <- list(
  "dataset_short_name" = "reanalysis-era5-land-monthly-means",
  "product_type"   = "monthly_averaged_reanalysis",
  "variable"       = "leaf_area_index_high_vegetation",
  "year"           = "2019",
  "month"          =  "06",
  "time"           = "00:00",
  "area"           = "70/-20/30/60",
  "format"         = "netcdf",
  "target"         = "demo_file.nc")

lai_hv_2019_06.nc  <- wf_request(user = "<your_ID>",
                     request = request_lai_hv_2019_06 ,
                     transfer = TRUE,
                     path = "data-raw",
                     verbose = FALSE)

Effective Leaf Area Index

You can find this data either in global computer raster images, or in re-processed monthly averages. Working with the raw data is not very practical – in case of cloudy weather you have missing data, and the files are extremely huge for a personal computer. For the purposes of our Green Deal Data Observatory the monthly average values are far more practical, which are called monthly_averaged_reanalysis product types.

For compatibility with other R packages, convert the data with the from raster package from rSpatial.org.

lai_file <- here::here( "data-raw", "demo_file.nc")
lai_raster <- raster::raster(lai_file)

## Loading required namespace: ncdf4

Let us convert this to a SpatialDataPointsDataFrame class, which is an augmented data frame class with coordinates.

LAI_df <- raster::rasterToPoints(lai_raster, fun=NULL, spatial=TRUE)

Get The Map

With the help fo rOpenGov, we are creating various R packages to programmatically access open data and put them into the right format. The popular eurostat package is not only useful to download data from Eurostat, but also to map it.

In this case, we want to create regional maps. Europe has five levels of geographical regions: NUTS0 for countries, NUTS1 for larger areas like states, provinces; NUTS2 for smaller areas like countries, NUTS3 for even smaller areas. The LAU level contains settlemens and their surrounding areas.

Country borders change sometimes (think about the unification of Germany, or the breakup of Czechoslovakia and Yugoslavia), but they are relatively stable entities. Sub-national regional border change very-very frequently – since 2000 there were many thousand changes in Europe. This means that you must choose one regional boundary definition. The latest edition is NUTS2021 but most of the data available is still in the NUTS2016 format, and often you will find NUTS2013 or even NUTS2010 data around. Our Green Deal Data Observatory uses the NUTS2016 definition, because it is far the most used in 2021. An offspring of the eurostat package, regions helps you take care of NUTS changes when you work, and can convert your data to NUTS2021 if you later need it.

## sf at resolution 1:60 read from local file

## Warning in eurostat::get_eurostat_geospatial(resolution = "60", nuts_level =
## "2", : Default of 'make_valid' for 'output_class="sf"' will be changed in the
## future (see function details).

plot(map_nuts_2)

Our measurement of the average Effective Leaf Area Index is a raster data, it is given for many points of Europe’s map. What we need to do is to overlay this raster information of the statistical map of Europe. We use the excellent sp: R Classes and Methods for Spatial Data package for this purpose. The sp::over() function decides if a point of Leaf Area Index measurement falls into the polygon (shape) of a particular NUTS2 regions, for example, Zuid-Holland or South Holland in the Netherlands, or Saarland in Germany, or not. Then it averages with the mean() function those measurements falling in the area.

LAI_nuts_2 = sp::over(sp::geometry(
  as(map_nuts_2, 'Spatial')), 
  LAI_df,
  fn=mean)

Let’s call the average LAI index lai, and bind it to the Eurostat map:

names(LAI_nuts_2)[1] <- "lai"
LAI_sfdf <- bind_cols ( map_nuts_2, LAI_nuts_2 )

If you want to work with the data in a numeric context, you do not need the geographical information, and you can “downgrade” the SpatialDataPointsDataFrame to a simple data frame.

set.seed(2019) #to always see the same sample
LAI_sfdf %>%
  as.data.frame() %>%
  select ( all_of(c("NUTS_NAME", "NUTS_ID", "lai")) ) %>%
  sample_n(10)

##                      NUTS_NAME NUTS_ID lai
## 281                       Vest    RO42  NA
## 125                     Kassel    DE73  NA
## 69              Friesland (NL)    NL12  NA
## 237 Agri, Kars, Igdir, Ardahan    TRA2  NA
## 273                East Anglia    UKH1  NA
## 119                Prov. Liège    BE33  NA
## 61                   Bourgogne    FRC1  NA
## 275                      Essex    UKH3  NA
## 282                   Istanbul    TR10  NA
## 174                    Leipzig    DED5  NA

We’ll plot the map with ggplot2.

library(ggplot2)
library(sf)
ggplot(data=LAI_sfdf) + 
  geom_sf(aes(fill=lai),
          color="dim grey", size=.1) + 
  scale_fill_gradient( low ="#FAE000", high = "#00843A") +
  guides(fill = guide_legend(reverse=T, title = "LAI")) +
  labs(title="Leaf Area Index",
       subtitle = "High vegetation half, NUTS2 regional avareage values",
       caption="\ua9 EuroGeographics for the administrative boundaries 
                \ua9 Copernicus Data Service, June 2019 average values
                Tutorial and ready-to-use data on greendeal.dataobservatory.eu") +
  theme_light() + theme(legend.position=c(.88,.78)) +
  coord_sf(xlim=c(-22,48), ylim=c(34,70))

Data Integrity

Our Green Deal Data Observatory has a data API where we place the new data with metadata for programmatic download in CSV, JSON or even with SQL queries. For data integrity purposes, we are placing an authoritative copy on Zenodo (Green Deal Data Observatory Community). You can use this for scientific citations. We are also happy if you place your own climate policy related research data here, so that we can include it in our observatory. In our subsequent tutorials, we will show how to do this programmatically in R. This particular dataset (not only with the month June, which we selected to streamline the tutorial) is available here with the digital object identifier doi.org/10.5281/zenodo.4903940.

Join us

Economic and Environment Impact Analysis, Automated for Data-as-Service

Thu, 03 Jun 2021 16:00:00 +0200

We have released a new version of iotables as part of the rOpenGov project. The package, as the name suggests, works with European symmetric input-output tables (SIOTs). SIOTs are among the most complex governmental statistical products. They show how each country’s 64 agricultural, industrial, service, and sometimes household sectors relate to each other. They are estimated from various components of the GDP, tax collection, at least every five years.

SIOTs offer great value to policy-makers and analysts to make more than educated guesses on how a million euros, pounds or Czech korunas spent on a certain sector will impact other sectors of the economy, employment or GDP. What happens when a bank starts to give new loans and advertise them? How is an increase in economic activity going to affect the amount of wages paid and and where will consumers most likely spend their wages? As the national economies begin to reopen after COVID-19 pandemic lockdowns, is to utilize SIOTs to calculate direct and indirect employment effects or value added effects of government grant programs to sectors such as cultural and creative industries or actors such as venues for performing arts, movie theaters, bars and restaurants.

Making such calculations requires a bit of matrix algebra, and understanding of input-output economics, direct, indirect effects, and multipliers. Economists, grant designers, policy makers have those skills, but until now, such calculations were either made in cumbersome Excel sheets, or proprietary software, as the key to these calculations is to keep vectors and matrices, which have at least one dimension of 64, perfectly aligned. We made this process reproducible with iotables and eurostat on rOpenGov

Our iotables package creates direct, indirect effects and multipliers programatically. Our observatory will make those indicators available for all European countries.

Accessing and tidying the data programmatically

The iotables package is in a way an extension to the eurostat R package, which provides a programmatic access to the Eurostat data warehouse. The reason for releasing a new package is that working with SIOTs requires plenty of meticulous data wrangling based on various metadata sources, apart from actually accessing the data itself. When working with matrix equations, the bar is higher than with tidy data. Not only your rows and columns must match, but their ordering must strictly conform the quadrants of the a matrix system, including the connecting trade or tax matrices.

When you download a country’s SIOT table, you receive a long form data frame, a very-very long one, which contains the matrix values and their labels like this:

## Table naio_10_cp1700 cached at C:\Users\...\Temp\RtmpGQF4gr/eurostat/naio_10_cp1700_date_code_FF.rds

# we save it for further reference here 
saveRDS(naio_10_cp1700, "not_included/naio_10_cp1700_date_code_FF.rds")

# should you need to retrieve the large tempfiles, they are in 
dir (file.path(tempdir(), "eurostat"))

dplyr::slice_head(naio_10_cp1700, n = 5)

## # A tibble: 5 x 7
##   unit    stk_flow induse  prod_na geo       time        values
##   <chr>   <chr>    <chr>   <chr>   <chr>     <date>       <dbl>
## 1 MIO_EUR DOM      CPA_A01 B1G     EA19      2019-01-01 141873.
## 2 MIO_EUR DOM      CPA_A01 B1G     EU27_2020 2019-01-01 174976.
## 3 MIO_EUR DOM      CPA_A01 B1G     EU28      2019-01-01 187814.
## 4 MIO_EUR DOM      CPA_A01 B2A3G   EA19      2019-01-01      0 
## 5 MIO_EUR DOM      CPA_A01 B2A3G   EU27_2020 2019-01-01      0

The metadata reads like this: the units are in millions of euros, we are analyzing domestic flows, and the national account items B1-B2 for the industry A01. The information of a 64x64 matrix (the SIOT) and its connecting matrices, such as taxes, or employment, or C**O₂ emissions, must be placed exactly in one correct ordering of columns and rows. Every single data wrangling error will usually lead in an error (the matrix equation has no solution), or, what is worse, in a very difficult to trace algebraic error. Our package not only labels this data meaningfully, but creates very tidy data frames that contain each necessary matrix of vector with a key column.

iotables package contains the vocabularies (abbreviations and human readable labels) of three statistical vocabularies: the so called COICOP product codes, the NACE industry codes, and the vocabulary of the ESA2010 definition of national accounts (which is the government equivalent of corporate accounting).

Our package currently solves all equations for direct, indirect effects, multipliers and inter-industry linkages. Backward linkages show what happens with the suppliers of an industry, such as catering or advertising in the case of music festivals, if the festivals reopen. The forward linkages show how much extra demand this creates for connecting services that treat festivals as a ‘supplier’, such as cultural tourism.

Let’s seen an example

## Downloading employment data from the Eurostat database.

## Table lfsq_egan22d cached at C:\Users\...\Temp\RtmpGQF4gr/eurostat/lfsq_egan22d_date_code_FF.rds

and match it with the latest structural information on from the Symmetric input-output table at basic prices (product by product) Eurostat product. A quick look at the Eurostat website already shows that there is a lot of work ahead to make the data look like an actual Symmetric input-output table. Download it with iotable_get() which does basic labelling and preprocessing on the raw Eurostat files. Because of the size of the unfiltered dataset on Eurostat, the following code may take several minutes to run.

sk_io <-  iotable_get ( labelled_io_data = NULL, 
                        source = "naio_10_cp1700", geo = "SK", 
                        year = 2015, unit = "MIO_EUR", 
                        stk_flow = "TOTAL",
                        labelling = "iotables" )

## Reading cache file C:\Users\..\Temp\RtmpGQF4gr/eurostat/naio_10_cp1700_date_code_FF.rds

## Table  naio_10_cp1700  read from cache file:  C:\Users\..\Temp\RtmpGQF4gr/eurostat/naio_10_cp1700_date_code_FF.rds

## Saving 808 input-output tables into the temporary directory
## C:\Users\...\Temp\RtmpGQF4gr

## Saved the raw data of this table type in temporary directory C:\Users\...\Temp\RtmpGQF4gr/naio_10_cp1700.rds.

The input_coefficient_matrix_create() creates the input coefficient matrix, which is used for most of the analytical functions.

a_i**j = X_i**j / x_j

It checks the correct ordering of columns, and furthermore it fills up 0 values with 0.000001 to avoid division with zero.

input_coeff_matrix_sk <- input_coefficient_matrix_create(
  data_table = sk_io
)

## Columns and rows of real_estate_imputed_a, extraterriorial_organizations are all zeros and will be removed.

Then you can create the Leontieff-inverse, which contains all the structural information about the relationships of 64x64 sectors of the chosen country, in this case, Slovakia, ready for the main equations of input-output economics.

I_sk <- leontieff_inverse_create(input_coeff_matrix_sk)

And take out the primary inputs:

primary_inputs_sk <- coefficient_matrix_create(
  data_table = sk_io, 
  total = 'output', 
  return = 'primary_inputs')

## Columns and rows of real_estate_imputed_a, extraterriorial_organizations are all zeros and will be removed.

Now let’s see if there the government tries to stimulate the economy in three sectors, agricultulre, car manufacturing, and R&D with a billion euros. Direct effects measure the initial, direct impact of the change in demand and supply for a product. When production goes up, it will create demand in all supply industries (backward linkages) and create opportunities in the industries that use the product themselves (forward linkages.)

direct_effects_create( primary_inputs_sk, I_sk ) %>%
  select ( all_of(c("iotables_row", "agriculture",
                    "motor_vechicles", "research_development"))) %>%
  filter (.data$iotables_row %in% c("gva_effect", "wages_salaries_effect", 
                                    "imports_effect", "output_effect"))

##            iotables_row agriculture motor_vechicles research_development
## 1        imports_effect   1.3684350       2.3028203            0.9764921
## 2 wages_salaries_effect   0.2713804       0.3183523            0.3828014
## 3            gva_effect   0.9669621       0.9790771            0.9669467
## 4         output_effect   2.2876287       3.9840251            2.2579634

Car manufacturing requires much imported components, so each extra demand will create a large importing activity. The R&D will create a the most local wages (and supports most jobs) because research is job-intensive. As we can see, the effect on imports, wages, gross value added (which will end up in the GDP) and output changes are very different in these three sectors.

This is not the total effect, because some of the increased production will translate into income, which in turn will be used to create further demand in all parts of the domestic economy. The total effect is characterized by multipliers.

Then solve for the multipliers:

multipliers_sk <- input_multipliers_create( 
  primary_inputs_sk %>%
    filter (.data$iotables_row == "gva"), I_sk )

And select a few industries:

set.seed(12)
multipliers_sk %>% 
  tidyr::pivot_longer ( -all_of("iotables_row"), 
                        names_to = "industry", 
                        values_to = "GVA_multiplier") %>%
  select (-all_of("iotables_row")) %>%
  arrange( -.data$GVA_multiplier) %>%
  dplyr::sample_n(8)

## # A tibble: 8 x 2
##   industry               GVA_multiplier
##   <chr>                           <dbl>
## 1 motor_vechicles                  7.81
## 2 wood_products                    2.27
## 3 mineral_products                 2.83
## 4 human_health                     1.53
## 5 post_courier                     2.23
## 6 sewage                           1.82
## 7 basic_metals                     4.16
## 8 real_estate_services_b           1.48

Vignettes

The Germany 1990 provides an introduction of input-output economics and re-creates the examples of the Eurostat Manual of Supply, Use and Input-Output Tables, by Jörg Beutel (Eurostat Manual).

The United Kingdom Input-Output Analytical Tables Daniel Antal, based on the work edited by Richard Wild is a use case on how to correctly import data from outside Eurostat (i.e., not with eurostat::get_eurostat()) and join it properly to a SIOT. We also used this example to create unit tests of our functions from a published, official government statistical release.

Finally, Working With Eurostat Data is a detailed use case of working with all the current functionalities of the package by comparing two economies, Czechia and Slovakia and guides you through a lot more examples than this short blogpost.

Our package was originally developed to calculate GVA and employment effects for the Slovak music industry, and similar calculations for the Hungarian film tax shelter. We can now programatically create reproducible multipliers for all European economies in the Digital Music Observatory, and create further indicators for economic policy making in the Economy Data Observatory.

Environmental Impact Analysis

Our package allows the calculation of various economic policy scenarios, such as changing the VAT on meat or effects of re-opening music festivals on aggregate demand, GDP, tax revenues, or employment. But what about the C**O₂, methane and other greenhouse gas effects of the reopening festivals, or the increasing meat prices?

Technically our package can already calculate such effects, but to do so, you have to carefully match further statistical vocabulary items used by the European Environmental Agency about air pollutants and greenhouse gases.

The last released version of iotables is Importing and Manipulating Symmetric Input-Output Tables (Version 0.4.4). Zenodo. https://doi.org/10.5281/zenodo.4897472, but we are already working on a new major release. (Download the BibLaTeX entry.) In that release, we are planning to build in the necessary vocabulary into the metadata functions to increase the functionality of the package, and create new indicators for our Green Deal Data Observatory. This experimental data observatory is creating new, high quality statistical indicators from open governmental and open science data sources that has not seen the daylight yet.

rOpenGov and the EU Datathon Challenges

rOpenGov, Reprex, and other open collaboration partners teamed up to build on our expertise of open source statistical software development further: we want to create a technologically and financially feasible data-as-service to put our reproducible research products into wider user for the business analyst, scientific researcher and evidence-based policy design communities.

rOpenGov is a community of open governmental data and statistics developers with many packages that make programmatic access and work with open data possible in the R language. Reprex is a Dutch-startup that teamed up with rOpenGov and other open collaboration partners to create a technologically and financially feasible service to exploit reproducible research products for the wider business, scientific and evidence-based policy design community. Open data is a legal concept - it means that you have the rigth to reuse the data, but often the reuse requires significant programming and statistical know-how. We entered into the annual EU Datathon competition in all three challenges with our applications to not only provide open-source software, but daily updated, validated, documented, high-quality statistical indicators as open data in an open database. Our iotables package is one of our many open-source building blocks to make open data more accessible to all.

Join our open collaboration Economy Data Observatory team as a data curator, developer or business developer. More interested in economic policies, particularly computation antitrust, innovation and small enterprises? Check out our Economy Music Observatory team! Or your interest lies more in data governance, trustworthy AI and other digital market problems? Check out our Digital Music Observatory team!

Data API

Tue, 01 Jun 2021 11:00:00 +0000

Our observatory has a new data API which allows access to our daily refreshing open data. You can access the API via api.greendeal.dataobservatory.eu

All the data and the metadata are available as open data, without database use restrictions, under the ODbL license. However, the metadata contents are fully not finalized yet. We are currently working on a solution that applies the FAIR Guiding Principles for scientific data management and stewardship, and fulfills the mandatory requirements of the Dublic Core metadata standards and at the same time the mandatory requirements, and most of the recommended requirements of DataCite.

Data table

The indicator table contains the actual values, and the various estimated/imputed values of the indicator, clearly marking missing values, too.

api.greendeal.dataobservatory.eu data retrieval

You can get the data in CSV or json format, or write SQL queries. (Tutorials in SQL, R, Python will be posted shortly.)

Descriptive metadata table

api.greendeal.dataobservatory.eu descriptive metadata

For further reference, see Descriptive Metadata.

Statistical Processing metadata table

api.greendeal.dataobservatory.eu processing metadata

For further reference, see Administrative (Processing) Metadata

Authoritative Copies

Greendeal Data Observatory on Zenodo

Metadata

Tue, 01 Jun 2021 11:00:00 +0000

Our observatory has a new data API which allows access to our daily refreshing open data. You can access the API via api.greendeal.dataobservatory.eu

All the data and the metadata are available as open data, without database use restrictions, under the ODbL license. However, the metadata contents are not finalized yet. We are currently working on a solution that applies the FAIR Guiding Principles for scientific data management and stewardship, and fulfills the mandatory requirements of the Dublic Core metadata standards and at the same time the mandatory requirements, and most of the recommended requirements of DataCite. These changes will be effective before 1 July 2021.

The Competition Data Observatory temporarily shares an API with the Economy Data Observatory, which serves as an incubator for similar economy-oriented reproducible research resources.

api.greendeal.dataobservatory.eu descriptive metadata

Descriptive Metadata


Identifier	An unambiguous reference to the resource within a given context. (Dublin Core item), but several identifiders allowed, and we will use several of them.
Creator	The main researchers involved in producing the data, or the authors of the publication, in priority order. To supply multiple creators, repeat this property. (Extends the Dublin Core with multiple authors, and legal persons, and adds affiliation data.)
Title	A name given to the resource. Extends Dublin Core with alternative title, subtitle, translated Title, and other title(s).
Publisher	The name of the entity that holds, archives, publishes prints, distributes, releases, issues, or produces the resource. This property will be used to formulate the citation, so consider the prominence of the role. For software, use Publisher for the code repository. (Dublin Core item.)
Publication Year	The year when the data was or will be made publicly available.
Resource Type	We publish Datasets, Images, Report, and Data Papers. (Dublin Core item with controlled vocabulary.)

Recommended for discovery

The Recommended (R) properties are optional, but strongly recommended for interoperability.


Subject	The topic of the resource. (Dublin Core item.)
Contributor	The institution or person responsible for collecting, managing, distributing, or otherwise contributing to the development of the resource. (Extends the Dublin Core with multiple authors, and legal persons, and adds affiliation data.) When applicable, we add Distributor (of the datasets and images), Contact Person, Data Collector, Data Curator, Data Manager, Hosting Institution, Producer (for images), Project Manager, Researcher, Research Group, Rightsholder, Sponsor, Supervisor
Date	A point or period of time associated with an event in the lifecycle of the resource, besides the Dublin Core minimum we add Collected, Created, Issued, Updated, and if necessary, Withdrawn dates to our datasets.
Related Identifier	An identifier or identifiers other than the primary Identifier applied to the resource being registered.
Rights	We give SPDX License List standards rights description with URLs to the actual license. (Dublin Core item: Rights Management)
Description	Recommended for discovery.(Dublin Core item.)
GeoLocation	Similar to Dublin Core item Coverage

The Subject property: we need to set standard coding schemas for each observatory.
Contributor property:
- DataCurator the curator of the dataset, who sets the mandatory properties.
- DataManager the person who keeps the dataset up-to-date.
- ContactPerson the person who can be contacted for reuse requests or bug reports.
The Date property contains the following dates, which are set automatically by the dataobservatory R package:
- Updated when the dataset was updated;
- EarliestObservation, which the earliest, not backcasted, estimated or imputed observation.
- LatestObservation, which the earliest, not backcasted, estimated or imputed observation.
- UpdatedatSource, when the raw data source was last updated.
The GeoLocation is automatically created by the dataobservatory R package.
The Description property optional elements, and we adopted them as follows for the observatories:
- The Abstract is a short, textual description; we try to automate its creation as much as a possible, but some curatorial input is necessary.
- In the TechnicalInfo sub-field, we record automatically the utils::sessionInfo() for computational reproducability. This is automatically created by the dataobservatory R package.
- In the Other sub-field, we record the keywords for structuring the observatory.

Optional

The Optional (O) properties are optional and provide richer description. For findability they are not so important, but to create a web service, they are essential. In the mandatory and recommended fields, we are following other metadata standards and codelists, but in the optional fields we have to build up our own system for the observatories.


Language	A language of the resource. (Dublin Core item.)
Alternative Identifier	An identifier or identifiers other than the primary Identifier applied to the resource being registered.
Size	We give the CSV, downloadable dataset size in bytes.
Format	We give file format information. We mainly use CSV and JSON, and occasionally rds and SPSS types. (Dublin Core item.)
Version	The version number of the resource.
Rights	We give SPDX License List standards rights description with URLs to the actual license. (Dublin Core item: Rights Management)
Funding Reference	We provide the funding reference information when applicable. This is usually mandatory with public funds.
Related Item	We give information about our observatory partners' related research products, awards, grants (also Dublin Core item as Relation.) We particularly include source information when the dataset is derived from another resource (which is a Dublin Core item.)

In the Language we only use English (eng) at the moment.
By default We do not use the Alternative Identifier property. We will do this when the same dataset will be used in several observatories.
The Size property is measured in bytes for the CSV representation of the dataset. During creations, the software creates a temporary CSV file to check if the dataset has no writing problems, and measures the dataset size.
The Version property needs further work. For a daily re-freshing API we need to find an applicable versioning system.
The Funding reference will contain information for donors, sponsors, and co-financing partners.
Our default setting for Rights is the CC-BY-NC-SA-4.0 license and we provide an URI for the license document.
In the RelatedItem we give information about:
- The original (raw) data source.
- Methodological bibilography reference, when needed.
- The open-source statistical software code that processed the data.

Administrative (Processing) Metadata

Like with diamonds, it is better to know the history of a dataset, too. Our administrative metadata contains codelists that follow the SXDX statistical metadata standards, and similarly strucutred information about the processing history of the dataset.

api.greendeal.dataobservatory.eu processing metadata

See for further reference The codebook Class.


Observation Status	SDMX Code list for Observation Status 2.2 (CL_OBS_STATUS), such as actual, missing, imputed, etc. values.
Method	If the value is estimated, we provide modelling information.
Unit	We provide the measurement unit of the data (when applicable.)
Frequency	SDMX Code list for Frequency 2.1 (CL_FREQ) frequency values
Codelist	Euros-SDMX Codelist entries for the observational units, such as sex, etc.
Imputation	SDMX Code list for Frequency 2.1 (CL_IMPUT_METH) imputation values
Estimation	The estimation methodology of data that we calculated, together with citation information and URI to the actual processing code
Related Item	We give information about the software code that processed the data (both Dublin Core and DataCite compliant.)

See an example in the The codebook Class article of the dataobservatory R package.

Data Sharing

Sun, 16 May 2021 00:00:00 +0000

we would like to actively encourage the sharing of data assets.

Open Data

Sun, 16 May 2021 00:00:00 +0000

Many countries in the world allow access to a vast array of information, such as documents under freedom of information requests, statistics, datasets. In the European Union, most taxpayer financed data in government administration, transport, or meteorology, for example, can be usually re-used. More and more scientific output is expected to be reviewable and reproducible, which implies open access.

What’s the Problem with Open Data?

How We Add Value?

Is There Value in It?
If it’s money on the street, why nobody’s picking it up?

Datasets Should Work Together to Give Information
Data is only potential information, raw and unprocessed.

What’s the Problem with Open Data?

“Data is stuff. It is raw, unprocessed, possibly even untouched by human hands, unviewed by human eyes, un-thought-about by human minds.” [1]

Most open data cannot be just “downloaded."
Often, you need to put more than $100 value of work into processing, validating, documenting a dataset that is worth $100. But you can share this investment with our data observatories.
Open data is almost always lacking of documentation, and no clear references to validate if the data is reliable or not corrupted. This is why we always start with reprocessing and redocumenting.

How We Add Value?

We believe that even such generally trusted data sources as Eurostat often need to be reprocessed, because various legal and political constraints do not allow the common European statistical services to provide optimal quality data – for example, on the regional and city levels.
With rOpenGov and other partners, we are creating open-source statistical software in R to re-process these heterogenous and low-quality data into tidy statistical indicators to automatically validate and document it.
Metadata is a potentially informative data record about a potentially informative dataset. We are carefully documenting and releasing administrative, processing, and descriptive metadata, following international metadata standards, to make our data easy to find and easy to use for data analysts.
We are automatically creating depositions and authoritative copies marked with an individual digital object identifier (DOI) to maintain data integrity.

Is There Value in Open Data?

A well-known story tells of a finance professor and a student who come across a $100 bill lying on the ground. As the student stops to pick it up, the professor says, “Don’t bother—if it were really a $100 bill, it wouldn’t be there.”

But this is not the case with open data. Often, you need to put more than $100 into processing, validating, documenting a dataset that is worth $100.

In the EU, open data is governed by the Directive on open data and the re-use of public sector information - in short: Open Data Directive (EU) 2019 / 1024. It entered into force on 16 July 2019. It replaces the Public Sector Information Directive, also known as the PSI Directive which dated from 2003 and was subsequently amended in 2013.

Open Data is potentially useful data that can potentially replace costlier or hard to get data sources to build information. It is analogous to potential energy: work is required to release it. We build automated systems that reduce this work and increase the likelihood that open data will offer the best value for money.

Most open data is not publicy accessible, and available upon request. Our real curatorial advantage is that we know where it is and how to get this request processed.
Most European open data comes from tax authorities, meteorological offices, managers of transport infrastructure, and other governmental bodies whose data needs are very different from yours. Their data must be carefully evaluated, re-processed, and if necessary, imputed to be usable for your scientific, business or policy goals.
The use of open science data is problematic in different ways: usually understanding the data documentation requires domain-specific specialist knowledge. Open science data is even more scattered and difficult to access than technically open, but not public governmental data.

From Datasets to Data Integration, Data to Information

“Data is only potential information, raw and unprocessed, prior to anyone actually being informed by it.” ^[2]

We are building simple databases and supporting APIs that release the data without restrictions, in a tidy format that is easy to join with other data, or easy to join into databases, together with standardized metadata.

Our service flow and value chain

FAQ

Why Downloading Does Not Work?

Most open data is not available on the internet.
If it is available, it is not in a form that you can easily import into a spreadsheet application like Excel or OpenOffice, or into a statistical application like SPSS or STATA.
Even the data quality of trusted web sources, like the Eurostat website, can be very low. Eurostat just publishes what it gets from governments, and often has no mandate to fix errors. The data is full with missing information, and in the case of regional statistics, faulty region codes and region names that make matching your data or placing them on a map impossible.
Adjusting euros with millions of euros, correctly translating dollars to euros, pounds to kilograms requires plenty of work. This is a very error-prone process when done by humans.

Can Open Data be Used in Machine Learning and AI?

Most public and open data sources have many missing observations; machine learning models usually cannot hanlde missingness. These points must be carefully imputed with approximations, which can be very challenging when the data has geographical dimension.
Removing missing values makes samples extremely biased and your model will learn from omissions, not information.

Photo Credits

What’s the Problem with Open Data? illustration is a photo by Cristina Gottardi How We Add Value? illustration is a photo by Nana Smirnova. Is There Value Left in It? is a photo by Imelda Datasets Should Work Together to Give Information is a photo by Lucas Santos

Footnote References

[1] Pomerantz, Jeffrey. 2021. “Metadata.” MIT Press essential knowledge series. MIT Press. Cambridge, Massachusetts ; London, England : The MIT Press, [2015]

[2] Pomerantz, Jeffrey. 2021. “Metadata.” MIT Press essential knowledge series. MIT Press. Cambridge, Massachusetts ; London, England : The MIT Press, [2015]

EU Datathon 2021

Wed, 12 May 2021 18:09:00 +0200

Reprex, a Dutch start-up enterprise formed to utilize open source software and open data, is looking for partners in an agile, open collaboration to win at least one of the three EU Datathon Prizes. We are looking for policy partners, academic partners and a consultancy partner. Our project is based on agile, open collaboration with three types of contributors.

With our competing prototypes we want to show that we have a research automation technology that can find open data, process it and validate it into high-quality business, policy or scientific indicators, and release it with daily refreshments in a modern API.

We are looking for institutions to challenge us with their data problems, and sponsors to increase our capacity. Over then next 5 months, we need to find a sustainable business model for a high-quality and open alternative to other public data programs.

The EU Datathon 2021 Challenge

To take part, you should propose the development of an application that links and uses open datasets. - our data curator team
Your application … is also expected to find suitable new approaches and solutions to help Europe achieve important goals set by the European Commission through the use of open data.” - this application is developed by our technology contributors
Your application should showcase opportunities for concrete business models or social enterprises. - our service development team is working to make this happen!
We use open source software and open data. The applications are hosted on the cloud resources of Reprex, an early-stage technology startup currently building a viable, open-source, open-data business model to create reproducible research products.
We are working together with experts in the domain as curators (check out our guidelines if you want to join: Data Curators: Get Inspired!).
Our development team works on an open collaboration basis. Our indicator R packages, and our services are developed together with rOpenGov.

Mission statement

We want to win an EU Datathon prize by processing the vast, already-available governmental and scientific open data made usable for policy-makers, scientific researchers, and business researcher end-users.

“To take part, you should propose the development of an application that links and uses open datasets. Your application should showcase opportunities for concrete business models or social enterprises. It is also expected to find suitable new approaches and solutions to help Europe achieve important goals set by the European Commission through the use of open data.”

We aim to win at least one first prize in the EU Datathon 2021. We are contesting all three challenges, which are related to the EU’s official strategic policies for the coming decade.

Challenge 1: A European Grean Deel

Our Green Deal Data Observatory connects socio-economic and environmental data to help understanding and combating climate change.

Challenge 1: A European Green Deal, with a particular focus on the The European Climate Pact, the Organic Action Plan, and the New European Bauhaus, i.e., mitigation strategies.

Climate change and environmental degradation are an existential threat to Europe and the world. To overcome these challenges, the European Union created the European Green Deal strategic plan, which aims to make the EU’s economy sustainable by turning climate and environmental challenges into opportunities and making the transition just and inclusive for all.

Our Green Deal Data Observatory is a modern reimagination of existing ‘data observatories’; currently, there are over 70 permanent international data collection and dissemination points. One of our objectives is to understand why the dozens of the EU’s observatories do not use open data and reproducible research. We want to show that open governmental data, open science, and reproducible research can lead to a higher quality and faster data ecosystem that fosters growth for policy, business, and academic data users.

We provide high quality, tidy data through a modern API which enables data flows between public and proprietary databases. We believe that introducing Open Policy Analysis standards with open data, open-source software, and research automation, can help the Green Deal policymaking process. Our collaboration is open for individuals, citizens scientists, research institutes, NGOS, and companies.

Challenge 2: An economy that works for people

Our Economy Data Observatory will focus on competition, small and medium sized enterprizes and robotization.

Challenge 2: An economy that works for people, with a particular focus on the Single market strategy, and particular attention to the strategy’s goals of 1. Modernising our standards system, 2. Consolidating Europe’s intellectual property framework, and 3. Enabling the balanced development of the collaborative economy strategic goals.

Big data and automation create new inequalities and injustices and have the potential to create a jobless growth economy. Our Economy Data Observatory is a fully automated, open source, open data observatory that produces new indicators from open data sources and experimental big data sources, with authoritative copies and a modern API.

Our observatory monitors the European economy to protect consumers and small companies from unfair competition, both from data and knowledge monopolization and robotization. We take a critical Small and Medium-Sized Enterprises (SME)-, intellectual property, and competition policy point of view of automation, robotization, and the AI revolution on the service-oriented European social market economy.

We would like to create early-warning, risk, economic effect, and impact indicators that can be used in scientific, business, and policy contexts for professionals who are working on re-setting the European economy after a devastating pandemic in the age of AI. We are particularly interested in designing indicators that can be early warnings for killer acquisitions, algorithmic and offline discrimination against consumers based on nationality or place of residence, and signs of undermining key economic and competition policy goals. Our goal is to help small and medium-sized enterprises and start-ups to grow, and to furnish data that encourages the financial sector to provide loans and equity funds for their growth.

Challenge 3: A Europe fit for the digital age

Our Digital Music Observatory is not only a demo of the European Music Observatory, but a testing ground for data governance, Digital Servcies Act, and trustworthy AI problems.

Challenge 3: A Europe fit for the digital age, with a particular focus Artificial Intelligence, the European Data Strategy, the Digital Services Act, Digital Skills and Connectivity.

The Digital Music Observatory (DMO) is a fully automated, open source, open data observatory that creates public datasets to provide a comprehensive view of the European music industry. It provides high-quality and timely indicators in all four pillars of the planned official European Music Observatory as a modern, open source and largely open data-based, automated, API-supported alternative solution for this planned observatory. The insight and methodologies we are refining in the DMO are applicable and transferable to about 60 other data observatories funded by the EU which do not currently employ governmental or scientific open data.

Music is one of the most data-driven service industries where most sales are currently executed by AI-driven autonomous systems that influence market shares and intellectual property remuneration. We provide a template that enables making these AI-driven systems accountable and trustworthy, with the goal of re-balancing the legitimate interests of creators, distributors, and consumers. Within Europe, this new balance will be an important use case of the European Data Strategy and the Digital Services Act.

The DMO is a fully functional service that can serve as a testing ground of the European Data Strategy. It can showcase the ways in which the music industry is affected by the problems that the Digital Services Act and European Trustworthy AI initiatives attempt to regulate. It is being built in open collaboration with national music stakeholders, NGOs, academic institutions, and industry groups.

Our Product/Market Fit was validated in the world’s 2nd ranked university-backed incubator program, the Yes!Delft AI Validation Lab. We are currently developing this project with the help of the JUMP European Music Market Accelerator program.

Problem Statement

The EU has an 18-year-old open data regime and it makes public taxpayer-funded data in the values of tens of billions of euros per year; the Eurostat program alone handles 20,000 international data products, including at least 5,000 pan-European environmental indicators.

As open science principles gain increased acceptance, scientific researchers are making hundreds of thousands of valuable datasets public and available for replication every year.

The EU, the OECD, and UN institutions run around 100 data collection programs, so-called ‘data observatories’ that more or less avoid touching this data, and buy proprietary data instead. Annually, each observatory spends between 50 thousand and 3 million EUR on collecting untidy and proprietary data of inconsistent quality, while never even considering open data.

Our automated data observatories are modern reimaginations of the existing observatories that do not use open data and research automation.

The problem with the current EU data strategy is that while it produces enormous quantities of valuable open data, in the absence of common basic data science and documentation principles, it seems often cheaper to create new data than to put the existing open data into shape.

This is an absolute waste of resources and efforts. With a few R packages and our deep understanding of advanced data science techniques, we can create valuable datasets from unprocessed open data. In most domains, we are able to repurpose data originally created for other purposes at a historical cost of several billions of euros, converting these unused data assets into valuable datasets that can replace tens of millions’ worth of proprietary data.

What we want to achieve with this project – and we believe such an accomplishment would merit one of the first prizes - is to add value to a significant portion of pre-existing EU open data (for example, available on data.europa.eu/data) by re-processing and integrating them into a modern, tidy database with an API access, and to find a business model that emphasises a triangular use of data in 1. business, 2. science and 3. policy-making. Our mission is to modernize the concept of data observatories.

Our solution

We are empowering data curators with reproducible research solutions to create high-quality, rigorously tested original datasets from low quality, not validated, not tidy open data. We help them to design meaningful business, policy or scientific indicators and provide them with a software and API to keep the data up-to-date. We help them deposit a copy of the authoritative, uncompromised dataset onto Zenodo, the EU’s data repository, with a DOI or new DOI version.
We create a research workflow that periodically (daily, weekly, monthly, quarterly or annually) collects, corrects and re-processes the data. We use peer-reviewed statistical software and unit-tests to make sure that the data is sound.

Panning out gold from muddy open sources - with automation technology.

We add value with correcting open (and proprietary!) data problems that make open data hard to use, and proprietary, in-house data hard to re-use.
- regions corrects inconsistent geographical coding. Eurostat has no mandate to correct geographical coding, and member states do not historically adjust their data. With many thousands of parish, county, region, province, state boundary changes within states, regional and metropolitian area datasets are not usable without our software.
- iotables puts extremely complex national accounts data into actually useful environmental and economic impact indicators. Instead of working with each country separately, our standardized system can calculate direct and indirect effects, as well as multipliers for every European country that works in the European statistical framework (EU member states, EEA, UK, member candidates.)
- retroharmonize connects cross-sectional surveys with non-European countries, puts pan-European surveys into time series, and corrects regional subsamples. We are creating new indicators from Eurobarometer, Afrobarometer, Arab Barometer, and standardized CAP surveys, as well as other harmonized surveys. We help design surveys that can utilize data from already existing, openly available surveys.
We place the authoritative copy to a data repository (Zenodo or Dataverse), automatically document the data, and make it available in a modern API for SQL queries or CSV downloads.
We present the data with commentary and blog posts from our curators (see: Is Drought Risk Uninsurable? - solidarity and climate change in Belgium) and contributors on a semi-automatically refreshed, open source web portal.
We are perfecting the agile open collaboration model in a triangular setting, where corporate users, scientific researchers, public and non-governmental policy makers, and even citizen scientists can work around a single data ecoystem.
We are validating a business model that allows the commercial, scientific, and policy use of re-processed, high quality data products made from open and shared data.

Recommendation Systems: What can Go Wrong with the Algorithm?

Thu, 06 May 2021 07:10:00 +0200

This is the edited text of my presentation on Copyright Data Improvement in the EU – Towards Better Visibility of European Content and Broader Licensing Opportunities in the Light of New Technologies - download the entire webinar’s agenda.

Assigning and avoding blame.

If you do not see enough women on streaming charts, or if you think that the percentage of European films on your favorite streaming provider—or Slovak music on your music streaming service—is too low, you have to be able to distribute the blame in more precise terms than just saying “it’s the system” that is stacked up against women, small countries, or other groups. We need to be able to point the blame more precisely in order to effect change through economic incentives or legal constraints.

This is precisely the type of work we are doing with the continued support of the Slovak national rightsholder organizations, as well as in our research in the United Kingdom. We try to understand why classical musicians are paid less, or why 15% of Slovak, Estonian, Dutch, and Hungarian artists never appear on anybody’s personalized recommendations. We need to understand how various AI-driven systems operate, and one approach would at the very least model and assign blame for undesirable outcomes in probabilistic terms. The problem is usually not that an algorithm is nasty and malicious; algorithms are often trained through “machine learning” techniques, and often, machines “learn” from biased, faulty, or low-quality information.

Outcomes: What Can Go Wrong With a Recommendation System?

In complex systems there are hardly ever singular causes that explain undesired outcomes; in the case of algorithmic bias in music streaming, there is no single bullet that eliminates women from charts or makes Slovak or Estonian language content less valuable than that in English. Some apparent causes may in fact be “blank cartridges,” and the real fire might come from unexpected directions. Systematic, robust approaches are needed in order to understand what it is that may be working against female or non-cisgender artists, long-tail works, or small-country repertoires.

Some examples of “undesirable outcomes” in recommendation engines might include:

Recommending too small a proportion of female or small country artists; or recommending artists that promote hate and violence.
Placing Slovak books on lower shelves.
Making the works of major labels easier to find than those of independent labels.
Placing a lower number of European works on your favorite video or music streaming platform’s start window than local television or radio regulations would require.
Filling up your social media newsfeed with fake news about covid-19 spread by some malevolent agents.

Metadata problems: no single bullet theory

In our work in Slovakia, we reverse engineered some of these undesirable outcomes. Popular video and music streaming recommendation systems have at least three major components based on machine learning:

The users’ history – Is it that users’ history is sexist, or perhaps the training metadata database is skewed against women?
The works’ characteristics – are Dvorak’s works as well documented for the algorithm as Taylor Swift’s or Drake’s?
Independent information from the internet – Does the internet write less about women artists?

In the making of a recommendation or an autonomous playlist, these sources of information can be seen as “metadata” concerning a copyright-protected work (as well as its right-protected recorded fixation.) More often than not, we are not facing a malicious algorithm when we see undesirable system outcomes. The usual problem is that the algorithm is learning from data that is historically biased against women or biased for British and American artists, or that it is only able to find data in English language film and music reviews. Metadata plays an incredibly important role in supporting or undermining general music education, media policy, copyright policy, or competition rules. If a video or music steaming platform’s algorithm is unaware of the music that music educators find suitable for Slovak or Estonian teenagers, then it will not recommend that music to your child.

Furthermore, metadata is very costly. In the case of cultural heritage, European states and the EU itself have been traditionally investing in metadata with each technological innovation. For Dvorak’s or Beethoven’s works, various library descriptions were made in the analogue world, then work and recording identifiers were assigned to CDs and mp3s, and eventually we must describe them again in a way intelligible for contemporary autonomous systems. In the case of classical music and literature, early cinema, or reproductions of artworks, we have public funding schemes for this work. But this seems not to be enough. In the current economy of streaming, the increasingly low income generated by most European works is insufficient to even cover the cost of proper documentation, which then sends that part of the European repertoire into a self-fulfilling oblivion: the algorithm cannot “learn” its properties and it never shows these works to users and audiences.

Until now, in most cases, it was assumed that it is the artists or their representative’s duty to provide high quality metadata, but in the analogue era, or in the era of individual digital copies, we did not anticipate that the sales value will not even cover the documentation cost. We must find technical solutions with interoperability and new economic incentives to create proper metadata for Europe’s cultural products. With that, we can cover one area out of the three possible problem terrains.

But this is not enough. We need to address the question of how new, better algorithms can learn from user history and avoid amplifying pre-existing bias against women or hateful speech. We need to make sure that when algorithms are “scraping” the internet, they do so in an accountable way that does not make small language repertoires vulnerable.

Incentives and investments into metadata

In our paper we argue for new regulatory considerations to create a better, and more accountable playing field for deploying algorithms in a quasi-autonomous system, and we suggest further research to align economic incentives with the creation of higher quality and less biased metadata. The need for further research on how these large systems affect various fundamental rights, consumer or competition rights, or cultural and media policy goals cannot be overstated. The first step is to open and understand these autonomous systems. It is not enough to say that the firing squads of Big Tech are shooting women out from charts, ethnic minority artists from screens, and small language authors from the virtual bookshelves. We must put a lot more effort on researching the sources of the problems that make machine learning algorithms behave in a way that is not compatible with our European values or regulations.

Is Drought Risk Uninsurable?

Fri, 23 Apr 2021 00:00:00 +0000

Climate change is real and it is everywhere. Whereas island nations in the Pacific are threatened with rising sea levels, Europe suffers from ever more frequent scorching summers and resulting drought. Take the case of Belgium, where heat waves in 2018 or 2020 have exacerbated an already fragile drought risk profile. An all too tangible effect is that houses built in areas where groundwater reservoirs are dwindling start to rupture. What adds insult to injury is that insurers appear unwilling to pay for damages: these climate-related risks simply did not feature in insurance policies made up decades ago. The public and the media have called upon the secretary of state responsible for consumer protection to come up with a solution. (Download this document in pdf.)

The Belgian insurance sector and government are currently investigating how to address the ecological and financial issue. Should the risk premium be raised on all insurance policies in an effort to spread risk, or should only policy holders in designated risk areas be subject to a raise in premia? Should urban planning initiatives and real estate projects be required to assess these new types of risk beforehand?

Driven by the Open Data Directive, we went in search for data at government websites such as waterinfo.be. That proved harder than you would want, with quite a number of technological barriers to cross. We independently explored the matter ourselves and came up with this: a dynamic map that pictures the spatial distribution of drought risk - as measured by a climate indicator known as the standardised recipitation-evapotranspiration index.

Actual drying soil.

This SPEI index, measured as a standardized variate, shows the deviations of the current climatic balance (precipitation minus evapotranspiration potential) in the long run and is presented on a monthly basis. As the SPEI in this form is more predictive for flood risk, we simply inverted the index to suggest a measure of drought risk¹.

Readers familiar with the “Kingdom by the sea” will remark that Belgium cannot possibly have a lack of precipitation. It rains more than the average Belgian cares for in the country. As a result, the water management system has historically been based on getting the water out as quickly as possible to the sea, in particular through the Ijzer, Schelde and Maas rivers. Add the abundance of concrete in the densely populated country - and its grossly mismanaged urban planning - and the capacity to hold water in surface and ground reservoirs is severely impaired. With climate change in full swing, these historical practices come back to haunt Belgium.

Are Belgians aware of climate risk?

We projected the public opinion data from Eurobarometer 90.2 (fieldwork: October-November 2018.) on the municipal map of Belgium. We used the answers to the multiple choice question QB1 Do you think that the following extreme weather events are due to climate change? We highlighted areas where people find it more likely to be exposed to Droughts and wildfires. We used the GESIS datafile (European Commission 2019) and used the (Antal 2021b, 2021a) packages to project the values to municipalities.

We see a weak spatial correlation between awareness of drought risk and actual draught risk. The least affected parts of Belgium appear least concerned. Despite its weakness, authorities and insurers can at least build their mitigation policies on a hypothesis of positive correlation. Of note is that concern for climate change effects follows regional, linguistic and other patterns. The map in particular suggests the Belgian provinces as markers for awareness.

Perception of likely drought.

Financial Capacity to Pay for Insurance

The next question we asked ourselves, was if the drought risk correlates with the ability to pay as distributed among local communities. Whether an insurance policy – or the regulation of insurance – attempts to provide cover on an individual level (through increased premia), or looks for local, regional or national mitigation strategies, the income/tax base might be an appropriate benchmark to test for financial capacity.

Financial capacity to mitigate drought risk.

The match between the (inverted) SPEI and total net income is less than perfect. Some of the areas most at risk coincide with the highest-income communities, but other threatened communities are low-income by Belgian standards. The actual risk awareness and the financial capacity to solve the problem are again only weakly correlated.^[2]

Correlation

Let’s have a look at the variables on NUTS3 level:

Correlation of the variables.

Average SPEI, which is a measure of increasing humidity, is negatively correlated with dry that we defined as -1 x avg_spei.
Dry areas, that are losing water, are less populous and more rich regions.
Dry_18 is a version of dry that only shows 12 months before the Eurobarometer survey about opinions on climate change effects, to see if the recent memory of actual weather conditions has had an affect of the perception of Belgians about these risk. It is seemingly not correlated with worries about floods or droughts.
Thedry_18 and the dry variables are largely correlated. One possible explanation is that the year before the survey was not an unusual period, it fit very well with the 2016-2020 trend.
Worries about extreme weather conditions are correlated with each other – i.e., some part of the population (concentrated geographically) is far more concerned with climate change than others.

The same on municipality (local administrative unit) level:

Correlation on the level of municipalities.

The correlations with opinion polling data are a little bit distorted, because the data is on NUTS2, and to bring it down to NUTS3 or LAU level would be a complicated small area statistical estimation task. We have also computed geospatial cross-correlation. Awareness of the climate problem and the dryness in 2018 were positively correlated in time – the drier the year was in an area, the more likely it was that people are aware of the problem; and the poorer areas were more likely to be afraid of this problem. The global spatial cross-correlation of the drying and local income was very low. This is a neutral situation: local income is not more concentrated to drying areas (which would be a lucky coincidence) nor concentrated in the relatively stable areas.

Generally, the problem map appears to be neutral to mildly favorable. The financial capacity to solve the problem is not working in the favor, nor against the problem, and awareness seems to be somewhat higher in the more affected areas.

The codes are in R/join_belgium_water_lau_dataset.R and R/join_belgium_water_nuts3_dataset.R.

Adverse Selection and Climate Solidarity

In addition to these historical analyses that put the drought risk in context, we are investigating whether climate data from integrated climate models might be harnessed to predict medium- to longer-term risk profiles on a spatially distributed basis. Urban planners, real estate promoters, individual households and governments will need to rely on such predictions to better adapt to climate change and reverse some of the earlier policy choices we mentioned.

To quote the Nobel Prize winning thoughts of Finn E. Kydland and Edward C. Prescott (Kydland and Prescott 1977):

The issues are obvious in many well-known problems of public policy. For example, suppose the socially desirable outcome is not to have houses built in a particular flood plain but, given that they are there, to take certain costly flood-control measures. If the government’s policy were not to build the dams and levees needed for flood protection and agents knew this was the case, even if houses were built there, rational agents would not live in the flood plains. But the rational agent knows that, if he and others build houses there, the government will take the necessary flood-control measures. Consequently, in the absence of a law prohibiting the construction of houses in the flood plain, houses are built there, and the army corps of engineers subsequently builds the dams and levees.

Our initial explorations at least suggest that leaving the resolution entirely to market forces, for example through increased property insurance premia may well lead to underinsurance in poorer areas that is dynamically inconsistent with government policy. If in particular severe drought will bankrupt farmers in such areas, eventually regional or national government will be forced to bail them out.

The other extreme approach, i.e., leaving the climate-change related damages entirely to the taxpayer, therefore does not seem feasible either with climate awareness and local income tax base only weakly correlating with the drought patterns. In addition, drought of course does not confine itself to municipal borders; the hydrological topology of the issue inherently implies a coordination problem between local, regional and federal entities passing the buck from one to another. One can imagine some form of solidarity and redistribution will be required to align interests and avoid adverse selection. To address these typical market failures, government will need to step in to allow these risks, that may be privately uninsurable, to be covered on a society-wide basis.

These problems are not unique to property damage. Similar problems arise in many student loan systems in the world (where it is desirable that the loan can be taken by arts students or future teachers, who may not have as high earning potential as easy-to-credit future lawyers, engineers, managers) or in many social security issues: a minimum level of health insurance for the unemployed and poor is desirable not only on the basis of humanity, but to avoid epidemic risks. Such special loan systems and special insurance systems are balancing some social welfare with individual welfare and individual risk considerations, and at the same time they try to avoid adverse selection, free-riding. We believe that our example can spark some ideas how a desirable social outcome can be aligned with the principles of insurance and personal responsibility.

In this case, on a longer term basis, incentives that may transfer water-intensive industrial and agricultural activities from the areas most at risk, could be called for, as well as better hydrological management to safeguard water reserves. We invite the authorities and relevant stakeholders to render the appropriate data needed to assess climate and drought evolution and to calculate risk premia scenarios and solidarity mechanisms open data, verified for quality through unit-tests and peer review.

References

Antal, Daniel. 2021a. Regions: Processing Regional Statistics. https://regions.danielantal.eu/.

———. 2021b. Retroharmonize: Ex Post Survey Data Harmonization. https://retroharmonize.dataobservatory.eu/.

Beguerı́a, Santiago, Sergio M Vicente-Serrano, Fergus Reig, and Borja Latorre. 2014. “Standardized Precipitation Evapotranspiration Index (SPEI) Revisited: Parameter Fitting, Evapotranspiration Models, Tools, Datasets and Drought Monitoring.” International Journal of Climatology 34 (10): 3001–23.

European Commission. 2019. “Eurobarometer 90.2 (2018).” GESIS Data Archive, Cologne. ZA7488 Data file Version 1.0.0, https://doi.org/10.4232/1.13289. https://doi.org/10.4232/1.13289.

Kydland, Finn E., and Edward C. Prescott. 1977. “Rules Rather Than Discretion: The Inconsistency of Optimal Plans.” Journal of Political Economy 85 (3): 473–91. http://www.jstor.org/stable/1830193.

Statbel. 2020. “Fiscal statistics on income.” Eurostat. https://statbel.fgov.be/en/open-data/fiscal-statistics-income.

Vicente-Serrano, Sergio M, Santiago Beguerı́a, and Juan I López-Moreno. 2010. “A Multiscalar Drought Index Sensitive to Global Warming: The Standardized Precipitation Evapotranspiration Index.” Journal of Climate 23 (7): 1696–1718.

As a standardized variate, SPEI can be compared across space and time. The original calculation of SPEI is based on the FAO-56 Penman-Monteith method. Other relevant indicators might consider the soil composition for example: clay and lime soils tend to be more vulnerable to drought. We combined this ecological dimension with the socio-economic dimension to suggest that insurance premia design might be targeted to, say, income levels as well - or alternatively to real estate prices. See (Beguerı́a et al. 2014; Vicente-Serrano, Beguerı́a, and López-Moreno 2010) ↩︎

Feasibility Study On Promoting Slovak Music In Slovakia & Abroad

Thu, 25 Mar 2021 11:00:00 +0100

How to help promote local music?

The new study opens the question of the local music promotion within the digital environment. The Slovak Performing and Mechanical Rights Society (SOZA), the State51 music group in the United Kingdom, and the Slovak Arts Council commissioned Reprex to created a feasibility study which provides recommendations for better use of quotas for Slovak radio stations and which also maps the share and promotion of Slovak music within large streaming and media platforms such as Spotify.

What should a good local content policy (radio quota, recommendation system, streaming quota) achieve?

The study proposes best practices for the introduction of mandatory quotas for Slovak radio stations and points out how current recommendation systems used by large platforms such as Spotify, YouTube, or Apple hardly consider local music from smaller countries. Local music stands against competition consisting of million songs from the whole world, and for ordinary Slovak musicians, whose music doesn’t belong to the global hits playlists, it is almost impossible to get recommended by the recommendation systems of large platforms.

Listen Local App for discovering new music

We aimed to create a demo version of a utility-based, transparent, accountable recommendation system.

The solution to this problem could be the Listen Local App, built on a comprehensive reference database of local music, which we created as a demo version within the study. The app aims to help listeners discover more local music; the app also presents new and alternative ways for large digital platforms to recommend local artists. Through Listen Local, listeners search for artists and bands based on their taste and the city they are situated in. In this way, listeners can easily search for music by artists from particular cities or from the town they are about to visit. We are releasing today the feasibility study in English and Slovak. We call for an open consultation to evaluate the results of this work and continue developing the Slovak Music Database, the Listen Local recommendation, and the AI validation system.

Check out the Demo Listen Local App. We explain here why.

Screenshot of the first verison of the demo app.

Database

The Slovak Music Database is connected to Reprex’s flagship project, the Demo Music Observatory, an open collaboration-based demo version of the planned European Music Observatory, currently being further developed in the JUMP Music Market Accelerator Programme supported by Music Moves Europe.

The project website contains the demo version of the Slovak Music Database.

Download the Study

You can download the study herein Slovak or in English.

Next steps

In the next phase of the work, we add further data to our Slovak Demo Music Database and carry out more and more experiments and educational activities to understand how Slovak music can become more visible and targeted. We are also bringing this project into an international collaboration for better utilization of R&D efforts and experiences throughout Europe. This agile project method originated in reproducible scientific practice and open-source software development and allows participation in large projects on any scale: from individual musicians and educators to large research universities and music distributors. Anyone can join in on the effort.

Reprex is looking for further international partners; Reprex is currently part of the Dutch AI Coalition and the European AI Alliance project. SOZA and Reprex are committed to opening this project for international collaboration while ensuring that a significant part of the R&D activities remains in the Slovak Republic.

We are preparing informal, online information sessions for artists, promoters, researchers, and developers to join our project.

Contributors

The Reprex team who contributed to the English version:

Budai, Sándor, programming and deployment
Dr. Emily H. Clarke, musicologist
Stef Koenis, musicologist, musician
Dr. Andrés Garcia Molina, data scientist, musicologist, editor
Kátya Nagy, music journalist, research assistant;

and the Slovak version:

Dáša Bulíková, musician, translator
Dominika Semaňáková, musicologist, editor, layout.

Special thanks to Tammy Nižňanska & the Youniverse for the case study.

Identifying Roadblocks to Net Zero Legislation

Tue, 16 Mar 2021 00:00:00 +0000

In our use case we are merging data about Europe’s coal regions, harmonized surveys about the acceptance of climate policies, and socio-economic data. While the work starts out from existing European research, our retroharmonize survey harmonization solution, our regions sub-national boundary harmonization solution and iotables allows us to connect open data and open knowledge from other coal regions of the world, for example, from the Appalachian economy.

Policy Context

The Just Transition Platform aims to assist EU countries and regions to unlock the support available through the Just Transition Mechanism. It builds on and expands the work of the existing Initiative for Coal Regions in Transition, which already supports fossil fuel producing regions across the EU in achieving a just transition through tailored, needs-oriented assistance and capacity-building.

The Initiative has a secretariat that is co-run by Ecorys, Climate Strategies, ICLEI Europe, and the Wuppertal Institute for Climate. While the initiative is an EU project, it cooperates with other similar initiatives, for example, with the Coalfield Development social enterprise in the Appalachian economy.

Data Sources

Coal regions: Our starting point is the EU coal regions: opportunities and challenges ahead publication Joint Research Centre (JRC), the European Commission’s science and knowledge service. This publication maps Europe’s coal dependent energy and transport infrastructure, and regions that depend on coal-related jobs.
Harmonized Survey Data: The dataset of the Eurobarometer 91.3 (April 2019) harmonized survey. Our transition policy variable is the four-level agreement with the statement More public financial support should be given to the transition to clean energies even if it means subsidies to fossil fuels should be reduced (EN) and Davantage de soutien financier public devrait être donné à la transition vers les énergies propres même si cela signifie que les subventions aux énergies fossiles devraient être réduites (FR) which is then translated to the language use of all participating country.
Environmental Variables: We used data on pm and SO2 polution measured by participating stations in the European Environmental Agency’s monitoring program. The station locations were mapped by Milos to the NUTS sub-national regions.

Exploratory Data Analysis

Our coal-dependency dummy variable is base on the policy document Coal regions in transition.

readRDS(file.path("data", "coal_regions.rds"))

## # A tibble: 253 x 5
##    country_code_is~ region_nuts_nam~ region_nuts_cod~ coal_region is_coal_region
##    <chr>            <fct>            <chr>            <chr>                <dbl>
##  1 BE               Brussels hoofds~ BE10             <NA>                     0
##  2 BE               Liege            BE33             <NA>                     0
##  3 BE               Brabant Wallon   BE31             <NA>                     0
##  4 BE               Antwerpen        BE21             <NA>                     0
##  5 BE               Limburg [BE]     BE22             <NA>                     0
##  6 BE               Oost-Vlaanderen  BE23             <NA>                     0
##  7 BE               Vlaams Brabant   BE24             <NA>                     0
##  8 BE               West-Vlaanderen  BE25             <NA>                     0
##  9 BE               Hainaut          BE32             <NA>                     0
## 10 BE               Namur            BE35             <NA>                     0
## # ... with 243 more rows

Our exploratory data analysis shows that respondent in 2019, agreement with the policy measure significantly differed among EU member states and regions.

transition_policy <- eb19_raw %>%
  rowid_to_column() %>%
  mutate ( transition_policy = normalize_text(transition_policy)) %>%
  fastDummies::dummy_cols(select_columns = 'transition_policy') %>%
  mutate ( transition_policy_agree = case_when(
    transition_policy_totally_agree + transition_policy_tend_to_agree > 0 ~ 1, 
    TRUE ~ 0
  )) %>%
  mutate ( transition_policy_disagree = case_when(
    transition_policy_totally_disagree + transition_policy_tend_to_disagree > 0 ~ 1, 
    TRUE ~ 0
  )) 

eb19_df  <- transition_policy %>% 
  left_join ( air_pollutants, by = 'region_nuts_codes' ) %>%
  mutate ( is_poland = ifelse ( country_code == "PL", 1, 0))

Preliminary Results

Significantly more people agree where

there are more polutants
who are younger
where people are more educated

Significantly less people agree

in rural areas
where more people are older
where more people are less educated
in less polluted areas
in coal regions

A simple model run:

c("transition_policy_totally_agree" , "pm10", "so2", "age_exact", "is_highly_educated" , "is_rural")

## [1] "transition_policy_totally_agree" "pm10"                           
## [3] "so2"                             "age_exact"                      
## [5] "is_highly_educated"              "is_rural"

summary( glm ( transition_policy_totally_agree ~ pm10 + so2 + 
                 age_exact +
                 is_highly_educated + is_rural + is_coal_region +
                 country_code, 
               data = eb19_df, 
               family = binomial ))

## 
## Call:
## glm(formula = transition_policy_totally_agree ~ pm10 + so2 + 
##     age_exact + is_highly_educated + is_rural + is_coal_region + 
##     country_code, family = binomial, data = eb19_df)
## 
## Deviance Residuals: 
##     Min       1Q   Median       3Q      Max  
## -1.7690  -1.0253  -0.8165   1.2264   1.9085  
## 
## Coefficients:
##                      Estimate Std. Error z value Pr(>|z|)    
## (Intercept)        -0.1975096  0.0921551  -2.143 0.032095 *  
## pm10                0.0068505  0.0017445   3.927 8.60e-05 ***
## so2                 0.1381994  0.0405867   3.405 0.000662 ***
## age_exact          -0.0075018  0.0007873  -9.529  < 2e-16 ***
## is_highly_educated  0.2953905  0.0311127   9.494  < 2e-16 ***
## is_rural           -0.1277983  0.0313321  -4.079 4.53e-05 ***
## is_coal_region     -0.2624005  0.0640233  -4.099 4.16e-05 ***
## country_codeBE     -0.3290891  0.0916117  -3.592 0.000328 ***
## country_codeBG     -0.6470116  0.1125114  -5.751 8.89e-09 ***
## country_codeCY      0.8471483  0.1273306   6.653 2.87e-11 ***
## country_codeCZ     -0.5754008  0.0965974  -5.957 2.57e-09 ***
## country_codeDE      0.0106430  0.0856322   0.124 0.901088    
## country_codeDK      0.0577724  0.0925391   0.624 0.532429    
## country_codeEE     -0.8041188  0.0989047  -8.130 4.28e-16 ***
## country_codeES      1.1266903  0.0941495  11.967  < 2e-16 ***
## country_codeFI     -0.2617501  0.0946837  -2.764 0.005702 ** 
## country_codeFR      0.0130239  0.1639339   0.079 0.936678    
## country_codeGB      0.2454631  0.0891845   2.752 0.005918 ** 
## country_codeGR      0.2169278  0.1209199   1.794 0.072816 .  
## country_codeHR     -0.1632727  0.1001563  -1.630 0.103064    
## country_codeHU      0.5779928  0.1020987   5.661 1.50e-08 ***
## country_codeIT     -0.1427249  0.0940144  -1.518 0.128985    
## country_codeLU     -0.3111627  0.1140426  -2.728 0.006363 ** 
## country_codeLV     -0.6246590  0.0963526  -6.483 8.99e-11 ***
## country_codeMT      0.3303363  0.1228611   2.689 0.007173 ** 
## country_codeNL      0.1707080  0.0902189   1.892 0.058470 .  
## country_codePL     -0.2843198  0.1228657  -2.314 0.020664 *  
## country_codePT      0.1447295  0.0899079   1.610 0.107452    
## country_codeRO     -0.0479674  0.0930433  -0.516 0.606177    
## country_codeSE      0.4865939  0.0922486   5.275 1.33e-07 ***
## country_codeSK     -0.2427307  0.0964652  -2.516 0.011861 *  
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## (Dispersion parameter for binomial family taken to be 1)
## 
##     Null deviance: 30568  on 22401  degrees of freedom
## Residual deviance: 29313  on 22371  degrees of freedom
##   (5253 observations deleted due to missingness)
## AIC: 29375
## 
## Number of Fisher Scoring iterations: 4

summary( glm ( transition_policy_agree ~ pm10 + so2 + age_exact +
                 is_highly_educated + is_rural, 
               data = eb19_df, 
               family = binomial ))

## 
## Call:
## glm(formula = transition_policy_agree ~ pm10 + so2 + age_exact + 
##     is_highly_educated + is_rural, family = binomial, data = eb19_df)
## 
## Deviance Residuals: 
##     Min       1Q   Median       3Q      Max  
## -2.1970   0.5035   0.5803   0.6495   0.8465  
## 
## Coefficients:
##                     Estimate Std. Error z value Pr(>|z|)    
## (Intercept)         1.807823   0.079297  22.798  < 2e-16 ***
## pm10                0.005092   0.001239   4.108 3.99e-05 ***
## so2                 0.003274   0.051410   0.064  0.94922    
## age_exact          -0.009781   0.000988  -9.900  < 2e-16 ***
## is_highly_educated  0.396743   0.039735   9.985  < 2e-16 ***
## is_rural           -0.107448   0.037953  -2.831  0.00464 ** 
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## (Dispersion parameter for binomial family taken to be 1)
## 
##     Null deviance: 20488  on 22401  degrees of freedom
## Residual deviance: 20250  on 22396  degrees of freedom
##   (5253 observations deleted due to missingness)
## AIC: 20262
## 
## Number of Fisher Scoring iterations: 4

Next Steps

After careful documentation, we will very soon publish all the processed, clean datasets on the EU Zenodo repository with clear digital object identification and versioning.
We will seek contact with the Secretariat of the Initiative for Coal Regions in Transition to process all the data annexes in the EU coal regions: opportunities and challenges ahead report.
With our volunteers we want to include coal regions from the United States, Latin America, Australia, Africa first – because we have harmonized survey results – and gradually add the rest of the world.
We will ask political scientists and policy researchers to interpret our findings.

Make Coal History

Tue, 16 Mar 2021 00:00:00 +0000

We find it difficult to imagine a path to net zero emissions that includes the use coal as a fuel. We would like to map communities in the world that still depend on coal mining.

Data curated by Milos

What?: We put coal mining areas on the global map? Why?: It allows us to connect coal mining to opinions about climate change and socio-economic indicators of communities. Use cases: Understanding how many people need an alternative to living from coal mining. Political dynamics of accepting climate change policies and individual climate action. Skills needed: Knowing where coal is mined in your country. How can you contribute:

Mapping Sub-national Boundaries

Tue, 16 Mar 2021 00:00:00 +0000

Working with sub-national data, on the level of provinces, states, regions, metropolitan areas brings far more insights into climate policies and potential actions than working with national data. The United States is a country, and so is Malta. China is a country, and so is Qatar. We believe that comparing Sydney Metropolitan Area with Paris Metropolitan Area, West Australia with Asturias in Spain is a far more fruitful approach.

Data curated by Daniel

What?: We want to integrate non-European sub-national boundaries, covered by the ISO 3166-2 standard into our regions package, similarly to Wikipedia and its sub-natioal maps. Why?: We can compare more homogeneous, more similar parts of the world with each other, we can compare social, economic, environmental variables and public opinion change in a far more relevant manner. Use cases: Understanding how many people need an alternative to living from coal mining. Political dynamics of accepting climate change policies and individual climate action. Skills needed: How can you contribute:

Finding out how can we parse data fairly from the ISO Online Browsing Platform
Writing the parsing code in R
Testing the parsing code in R
Writing tutorials
Finding ways to connect ISO 3166-2 sub-national boundaries with other sub-national boundaries, such as Europe’s NUTS.

Reprex Open Data Day 2021

Sat, 06 Mar 2021 15:30:00 +0200

Open Data Day is an annual celebration of open data all over the world. It is an opportunity to show the benefits of open data and encourage the adoption of open data policies in government, business, and civil society. Reprex is a start-up that utilizes open data with open-source reproducible research: please challenge us with your data requests and participate in our web events.

The Reprex Open Data Day 2021 will be two informal conversations based on a series of run up introductory blogposts centered around two themes. Because important guests became ill in the last days, we are going to consolidate the two talks into one with less structure. We want to create an informal, inclusive, collaborative online event on International Open Data Day 2021. Please, grab a tea, coffee, or even a beer, and join us for an informal conversation. We hope that we will finish the afternoon with ideas on new, open-data driven collaborations.

9.30 EST / 15.30 CET: Open collaboration in business, policy and science. Creating evidence-based policy, business strategy or scientific research with small contributions with independent components with incentives. Short introduction with examples: joining environmental sensory data and public opinion data on maps; creating harmonized datasets across the Arab world. Survey harmonization, mapping, data products. Scaling up open collaboration: making small organizations competitive with big tech in the big data era. Data sharing, data pooling, data altruism and observatories. The new European trustworthy AI and data governance agenda.

You can click through a short presentation to familiarize yourself with our topics.

See you here.

Case studies:

We are connecting raw survey data about Climate Awareness in Eurobarometer surveys. Here is the reproduction code (intermediate to advanced R needed.) You should use the development version of our retroharmonize package at github.com/antaldaniel/retroharmonize
We are tracking changes in the boundaries of provinces, states, counties, parishes with our regions open source software – reproduction code here. You will need our regions package which is available on CRAN or in the rOpenGov GitHub repo.
We will talk about how to join this with air pollution data and put it on the map with Milos Popovic, who prepared this nice choropleth animation.

We will discuss data observatories (permanent data collection programs), open collaboration (open-source inspired way of cooperation among small and large independent actors) and data altruism.

Any questions: send Daniel a message on Keybase, Whatsapp or email.

Hello on International #OpenDataDay2021 from🌷 the Hague!
- We have brought some new data to the light about 🌡climate change awareness
- We created some tutorials how to harmonize survey and geographical data
- Join us at 9.30 EST/15.30 CET 👇https://t.co/7J7pvi3sPC #ODD2021 pic.twitter.com/DwkGQaDhW1
— dataandlyrics (@dataandlyrics) March 6, 2021

Reprex Open Data Day 2021

Sat, 06 Mar 2021 15:30:00 +0200

You can click through a short presentation to familiarize yourself with our topics.

See you here.

Case studies:

We are connecting raw survey data about Climate Awareness in Eurobarometer surveys. Here is the reproduction code (intermediate to advanced R needed.) You should use the development version of our retroharmonize package at github.com/antaldaniel/retroharmonize
We are tracking changes in the boundaries of provinces, states, counties, parishes with our regions open source software – reproduction code here. You will need our regions package which is available on CRAN or in the rOpenGov GitHub repo.
We will talk about how to join this with air pollution data and put it on the map with Milos Popovic, who prepared this nice choropleth animation.

We will discuss data observatories (permanent data collection programs), open collaboration (open-source inspired way of cooperation among small and large independent actors) and data altruism.

Any questions: send Daniel a message on Keybase, Whatsapp or email.

Hello on International #OpenDataDay2021 from🌷 the Hague!
- We have brought some new data to the light about 🌡climate change awareness
- We created some tutorials how to harmonize survey and geographical data
- Join us at 9.30 EST/15.30 CET 👇https://t.co/7J7pvi3sPC #ODD2021 pic.twitter.com/DwkGQaDhW1
— dataandlyrics (@dataandlyrics) March 6, 2021

Regional Geocoding Harmonization Case Study - Regional Climate Change Awareness Datasets

Sat, 06 Mar 2021 00:00:00 +0000

library(regions)
library(lubridate)
library(dplyr)

if ( dir.exists('data-raw') ) {
  data_raw_dir <- "data-raw"
} else {
  data_raw_dir <- file.path("..", "..", "data-raw")
  }

Going beyond the national level

Let’s start with a dirty averaging by sub-national unit. The w1 weighting variable contains the post-stratification weight for the national samples. The Eurobarometer samples represent nations (with the exception of East and West Germany, Northern Ireland and Great Britain.) The average of the w1 variable is 1.00 for each sample, but it is not necessarily 1 for smaller territorial units. If sum(w)>1 for say, AT23 it only means that the AT23 region was undersampled relatively to the rest of Austria, and responses must be over-weighted in post-stratification.

There is no way to make the samples become regionally representative, and a correct post-stratification would require further data about the sampel design. But we can simply adjust to over/undersampling by making sure that oversampled territorial averages are proportionally increased and undersampled ones are decreased. [Another ‘dirty’ averaging would be the use of an unweighted average, but our method is better, because it more-or-less adjusts gender and education level biases, but leaves intra-country regional biases in the sample.]

panel <- readRDS((file.path(data_raw_dir, "climate-panel.rds")))

climate_data <-  panel %>%
  mutate ( year = lubridate::year(date_of_interview)) %>%
  select ( all_of(c("isocntry", "geo", "w1")), 
           contains("problem")
  )  %>%
  mutate ( 
    # use the post-stratification weights for national samples
    serious_world_problems_first = w1*serious_world_problems_first , 
    serious_world_problems_climate_change = w1*serious_world_problems_climate_change) %>%
  group_by (  .data$geo ) %>%
  summarise( serious_world_problems_first = mean(serious_world_problems_first, na.rm=TRUE),
             serious_world_problems_climate_change = mean (serious_world_problems_climate_change, na.rm=TRUE),
             mean_w1 = mean(w1)
             ) %>%
  mutate ( 
    # adjust for post-stratification weight bias due to regional over/undersampling
    climate_first = serious_world_problems_first / mean_w1, 
    climate_mentioned = serious_world_problems_climate_change / mean_w1
    )

So, we averaged, weighted and adjusted the mentioning of climate change as the world’s most serious, or one of the most serious problems by NUTS regions.

Aggregation level

The problem is that most statistical data is available in for the NUTS regional boundaries according to the NUTS2016 definition. However, GESIS uses NUTS2013 regions, so 252 regional codes in the four survey waves are invalid. Some data is available only on national level, but it can be projected to regional level, because small countries like Luxembourg have no regional divisions. Larger countries like Germany are divided only on state level (NUTS1), while small countries are divided on NUTS3 level.

This leads to various problems. Many data is available only on NUTS2 level, in which case NUTS1 data should be projected to its constituent smaller NUTS2 regions, and NUTS3 level data must be aggregated up to larger, containing NUTS2 levels.

Of course, we also must choose if we use `NUTS2013 or NUTS2016 boundaries. Sub-national boundaries have changed many thousand times in the EU27 countries alone since 1999.

## # A tibble: 5 x 2
##   validate         n
##   <chr>        <int>
## 1 country         15
## 2 invalid        252
## 3 nuts_level_1   132
## 4 nuts_level_2   452
## 5 nuts_level_3   141

Recoding the Regions

Our regions package was designed to keep track of sub-national regional boundary changes. It can validate regional data codes, and to some extent carry out recoding, imputation or simple aggregation.

Recoding means that the boundaries are unchanged, but the country changed the names/codes of regions, because there were other boundary changes which did not affect our observation unit.
Imputation must not be done with usual, general imputation tools, because our data is regionally structured. However, some imputations are very simple, because we can use equality equasions like MT = MT0, MT00.
Often the boundary change is additive, and merged territorial units can simple aggregated for comparison in earlier data.

regional_coding_2016 <- panel %>%
  mutate ( year = lubridate::year(date_of_interview)) %>%
  select (  all_of(c("isocntry", "geo", "region", "year") ) ) %>%
  distinct_all() %>%
  recode_nuts()

regional_coding_2013 <- panel %>%
  mutate ( year = lubridate::year(date_of_interview)) %>%
  select (  all_of(c("isocntry", "geo", "region", "year") ) ) %>%
  distinct_all() %>%
  recode_nuts( nuts_year = 2013)

climate_data_recoded <- climate_data %>% 
  left_join ( regional_coding_2016, by = 'geo' ) %>%
  left_join ( regional_coding_2013 %>% 
                select ( all_of(c("geo", "code_2013"))), 
              by = "geo") %>%
  distinct_all()

saveRDS ( climate_data_recoded , file.path(tempdir(), "climate_panel_recoded_agr.rds"), version = 2)

# not evaluated
saveRDS( climate_data_recoded , file = file.path("data-raw", "climate_panel_recoded_agr.rds"))

Where Are People More Likely To Treat Climate Change as the Most Serious Global Problem?

Sat, 06 Mar 2021 00:00:00 +0000

library(regions)
library(lubridate)
library(dplyr)

if ( dir.exists('data-raw') ) {
  data_raw_dir <- "data-raw"
} else {
  data_raw_dir <- file.path("..", "..", "data-raw")
  }

The first results of our longitudinal table were difficult to map, because the surveys used an obsolete regional coding. We will adjust the wrong coding, when possible, and join the data with the European Environment Agency’s (EEA) Air Quality e-Reporting (AQ e-Reporting) data on environmental pollution. We recoded the annual level for every available reporting stations [not shown here] and all values are in μg/m3. The period under observation is 2014-2016. Data file: https://www.eea.europa.eu/data-and-maps/data/aqereporting-8 (European Environment Agency 2021).

Recoding the Regions

Recoding means that the boundaries are unchanged, but the country changed the names and codes of regions because there were other boundary changes which did not affect our observation unit. We explain the problem and the solution in greater detail in our tutorial that aggregates the data on regional levels.

panel <- readRDS((file.path(data_raw_dir, "climate-panel.rds")))

climate_data_geocode <-  panel %>%
  mutate ( year = lubridate::year(date_of_interview)) %>%
  recode_nuts()

Let’s join the air pollution data and join it by corrected geocodes:

load(file.path("data", "air_pollutants.rda")) ## good practice to use system-independent file.path

climate_awareness_air <- climate_data_geocode %>%
  rename ( region_nuts_codes  = .data$code_2016) %>%
  left_join ( air_pollutants, by = "region_nuts_codes" ) %>%
  select ( -all_of(c("w1", "wex", "date_of_interview", 
                     "typology", "typology_change", "geo", "region"))) %>%
  mutate (
    # remove special labels and create NA_numeric_ 
    age_education = retroharmonize::as_numeric(age_education)) %>%
  mutate_if ( is.character, as.factor) %>%
  mutate ( 
    # we only have responses from 4 years, and this should be treated as a categorical variable
    year = as.factor(year) 
    ) %>%
  filter ( complete.cases(.) )

The climate_awareness_air data frame contains the answers of 75086 individual respondents. 17.07% thought that climate change was the most serious world problem and 33.6% mentioned climate change as one of the three most important global problems.

summary ( climate_awareness_air  )

##                  rowid       serious_world_problems_first
##  ZA5877_v2-0-0_1    :    1   Min.   :0.0000              
##  ZA5877_v2-0-0_10   :    1   1st Qu.:0.0000              
##  ZA5877_v2-0-0_100  :    1   Median :0.0000              
##  ZA5877_v2-0-0_1000 :    1   Mean   :0.1707              
##  ZA5877_v2-0-0_10000:    1   3rd Qu.:0.0000              
##  ZA5877_v2-0-0_10001:    1   Max.   :1.0000              
##  (Other)            :75080                               
##  serious_world_problems_climate_change    isocntry    
##  Min.   :0.000                         BE     : 3028  
##  1st Qu.:0.000                         CZ     : 3023  
##  Median :0.000                         NL     : 3019  
##  Mean   :0.336                         SK     : 3000  
##  3rd Qu.:1.000                         SE     : 2980  
##  Max.   :1.000                         DE-W   : 2978  
##                                        (Other):57058  
##                                    marital_status         age_education  
##  (Re-)Married: without children           :13242   18            :15485  
##  (Re-)Married: children this marriage     :12696   19            : 7728  
##  Single: without children                 : 7650   16            : 5840  
##  (Re-)Married: w children of this marriage: 6520   still studying: 5098  
##  (Re-)Married: living without children    : 6225   17            : 5092  
##  Single: living without children          : 4102   15            : 4528  
##  (Other)                                  :24651   (Other)       :31315  
##    age_exact                      occupation_of_respondent
##  Min.   :15.0   Retired, unable to work       :22911      
##  1st Qu.:36.0   Skilled manual worker         : 6774      
##  Median :51.0   Employed position, at desk    : 6716      
##  Mean   :50.1   Employed position, service job: 5624      
##  3rd Qu.:65.0   Middle management, etc.       : 5252      
##  Max.   :99.0   Student                       : 5098      
##                 (Other)                       :22711      
##             occupation_of_respondent_recoded
##  Employed (10-18 in d15a)   :32763          
##  Not working (1-4 in d15a)  :37125          
##  Self-employed (5-9 in d15a): 5198          
##                                             
##                                             
##                                             
##                                             
##                        respondent_occupation_scale_c_14
##  Retired (4 in d15a)                   :22911          
##  Manual workers (15 to 18 in d15a)     :15269          
##  Other white collars (13 or 14 in d15a): 9203          
##  Managers (10 to 12 in d15a)           : 8291          
##  Self-employed (5 to 9 in d15a)        : 5198          
##  Students (2 in d15a)                  : 5098          
##  (Other)                               : 9116          
##                   type_of_community   is_student      no_education     
##  DK                        :   34   Min.   :0.0000   Min.   :0.000000  
##  Large town                :20939   1st Qu.:0.0000   1st Qu.:0.000000  
##  Rural area or village     :24686   Median :0.0000   Median :0.000000  
##  Small or middle sized town: 9850   Mean   :0.0679   Mean   :0.008151  
##  Small/middle town         :19577   3rd Qu.:0.0000   3rd Qu.:0.000000  
##                                     Max.   :1.0000   Max.   :1.000000  
##                                                                        
##    education       year       region_nuts_codes  country_code  
##  Min.   :14.00   2013:25103   LU     : 1432     DE     : 4531  
##  1st Qu.:17.00   2015:    0   MT     : 1398     GB     : 3538  
##  Median :18.00   2017:25053   CY     : 1192     BE     : 3028  
##  Mean   :19.61   2019:24930   SK02   : 1053     CZ     : 3023  
##  3rd Qu.:22.00                EL30   :  974     NL     : 3019  
##  Max.   :30.00                EE     :  973     SK     : 3000  
##                               (Other):68064     (Other):54947  
##      pm2_5             pm10               o3              BaP        
##  Min.   : 2.109   Min.   :  5.883   Min.   : 66.37   Min.   :0.0102  
##  1st Qu.: 9.374   1st Qu.: 28.326   1st Qu.: 90.89   1st Qu.:0.1779  
##  Median :11.866   Median : 33.673   Median :102.81   Median :0.4105  
##  Mean   :12.954   Mean   : 38.637   Mean   :101.49   Mean   :0.8759  
##  3rd Qu.:15.890   3rd Qu.: 49.488   3rd Qu.:110.73   3rd Qu.:1.0692  
##  Max.   :41.293   Max.   :123.239   Max.   :141.04   Max.   :7.8050  
##                                                                      
##       so2              ap_pc1            ap_pc2             ap_pc3       
##  Min.   : 0.0000   Min.   :-4.6669   Min.   :-2.21851   Min.   :-2.1007  
##  1st Qu.: 0.0000   1st Qu.:-0.4624   1st Qu.:-0.49130   1st Qu.:-0.5695  
##  Median : 0.0000   Median : 0.4263   Median : 0.02902   Median :-0.1113  
##  Mean   : 0.1032   Mean   : 0.1031   Mean   : 0.04166   Mean   :-0.1746  
##  3rd Qu.: 0.0000   3rd Qu.: 0.9748   3rd Qu.: 0.57416   3rd Qu.: 0.3309  
##  Max.   :42.5325   Max.   : 2.0344   Max.   : 3.25841   Max.   : 4.1615  
##                                                                          
##      ap_pc4            ap_pc5        
##  Min.   :-1.7387   Min.   :-2.75079  
##  1st Qu.:-0.1669   1st Qu.:-0.18748  
##  Median : 0.0371   Median : 0.01811  
##  Mean   : 0.1154   Mean   : 0.06797  
##  3rd Qu.: 0.3050   3rd Qu.: 0.34937  
##  Max.   : 3.2476   Max.   : 1.42816  
##

Let’s see a simple CART tree! We remove the regional codes, because there are very serious differences among regional climate awareness. These differences, together with education level, and the year we are talking about, are the most important predictors of thinking about climate change as the most important global problem in Europe.

# Classification Tree with rpart
library(rpart)

# grow tree
fit <- rpart(as.factor(serious_world_problems_first) ~ .,
   method="class", data=climate_awareness_air %>%
     select ( - all_of(c("rowid", "region_nuts_codes"))), 
   control = rpart.control(cp = 0.005))

printcp(fit) # display the results

## 
## Classification tree:
## rpart(formula = as.factor(serious_world_problems_first) ~ ., 
##     data = climate_awareness_air %>% select(-all_of(c("rowid", 
##         "region_nuts_codes"))), method = "class", control = rpart.control(cp = 0.005))
## 
## Variables actually used in tree construction:
## [1] age_education                         isocntry                             
## [3] serious_world_problems_climate_change year                                 
## 
## Root node error: 12817/75086 = 0.1707
## 
## n= 75086 
## 
##          CP nsplit rel error  xerror      xstd
## 1 0.0240566      0   1.00000 1.00000 0.0080438
## 2 0.0082703      3   0.92783 0.92783 0.0078055
## 3 0.0050000      5   0.91129 0.91425 0.0077588

plotcp(fit) # visualize cross-validation results

summary(fit) # detailed summary of splits

## Call:
## rpart(formula = as.factor(serious_world_problems_first) ~ ., 
##     data = climate_awareness_air %>% select(-all_of(c("rowid", 
##         "region_nuts_codes"))), method = "class", control = rpart.control(cp = 0.005))
##   n= 75086 
## 
##            CP nsplit rel error    xerror        xstd
## 1 0.024056592      0 1.0000000 1.0000000 0.008043837
## 2 0.008270266      3 0.9278302 0.9278302 0.007805478
## 3 0.005000000      5 0.9112897 0.9142545 0.007758824
## 
## Variable importance
## serious_world_problems_climate_change                              isocntry 
##                                    31                                    26 
##                          country_code                                   BaP 
##                                    20                                     8 
##                                 pm2_5                                ap_pc1 
##                                     4                                     3 
##                         age_education                                  pm10 
##                                     2                                     2 
##                             education                                ap_pc2 
##                                     2                                     1 
##                                  year 
##                                     1 
## 
## Node number 1: 75086 observations,    complexity param=0.02405659
##   predicted class=0  expected loss=0.1706976  P(node) =1
##     class counts: 62269 12817
##    probabilities: 0.829 0.171 
##   left son=2 (25229 obs) right son=3 (49857 obs)
##   Primary splits:
##       serious_world_problems_climate_change < 0.5          to the right, improve=2214.2040, (0 missing)
##       isocntry                              splits as  RRLLLRRRLLRLRLLLLLLLLLLRRLLLRLL, improve= 728.0160, (0 missing)
##       country_code                          splits as  RRLLLRRLLRLLLLLLLLLLRRLLLRLL, improve= 673.3656, (0 missing)
##       BaP                                   < 0.4300347    to the right, improve= 310.6229, (0 missing)
##       pm2_5                                 < 13.38264     to the right, improve= 296.4013, (0 missing)
##   Surrogate splits:
##       age_education splits as  ----RRRRRR-RRRRRRRRRR-RRRRRRRRRR-RRRRRRRRRR-RRRRRRRRRR-RRRRRL-RRR-RRRRRRRRR--RRRLLR--R-R, agree=0.664, adj=0, (0 split)
##       pm10          < 7.491315     to the left,  agree=0.664, adj=0, (0 split)
## 
## Node number 2: 25229 observations
##   predicted class=0  expected loss=0  P(node) =0.3360014
##     class counts: 25229     0
##    probabilities: 1.000 0.000 
## 
## Node number 3: 49857 observations,    complexity param=0.02405659
##   predicted class=0  expected loss=0.2570752  P(node) =0.6639986
##     class counts: 37040 12817
##    probabilities: 0.743 0.257 
##   left son=6 (34631 obs) right son=7 (15226 obs)
##   Primary splits:
##       isocntry     splits as  RRLLLRRRLLRLRLLLLLLLLLLRRLLLRLL, improve=1454.9460, (0 missing)
##       country_code splits as  RRLLLRRLLRLLLLLLLLLLRRLLLRLL, improve=1359.7210, (0 missing)
##       BaP          < 0.4300347    to the right, improve= 629.8844, (0 missing)
##       pm2_5        < 13.38264     to the right, improve= 555.7484, (0 missing)
##       ap_pc1       < -0.005459537 to the left,  improve= 533.3579, (0 missing)
##   Surrogate splits:
##       country_code splits as  RRLLLRRLLRLLLLLLLLLLRRLLLRLL, agree=0.987, adj=0.957, (0 split)
##       BaP          < 0.1749425    to the right, agree=0.775, adj=0.264, (0 split)
##       pm2_5        < 5.206993     to the right, agree=0.737, adj=0.140, (0 split)
##       ap_pc1       < 1.405527     to the left,  agree=0.733, adj=0.126, (0 split)
##       pm10         < 25.31211     to the right, agree=0.718, adj=0.076, (0 split)
## 
## Node number 6: 34631 observations
##   predicted class=0  expected loss=0.1769802  P(node) =0.4612178
##     class counts: 28502  6129
##    probabilities: 0.823 0.177 
## 
## Node number 7: 15226 observations,    complexity param=0.02405659
##   predicted class=0  expected loss=0.4392487  P(node) =0.2027808
##     class counts:  8538  6688
##    probabilities: 0.561 0.439 
##   left son=14 (11607 obs) right son=15 (3619 obs)
##   Primary splits:
##       isocntry      splits as  LL---LLR--L-L----------LL---R--, improve=337.5462, (0 missing)
##       country_code  splits as  LL---LR--L-L--------LL---R--, improve=337.5462, (0 missing)
##       age_education splits as  ----LLLLLL-LLLRRRRRRR-RRRRRRRRRL-RRRRRRLLRR-RRRRLLRLRL-RRLRRR-RRR-LLLLRRR-----LR-----L-R, improve=294.0807, (0 missing)
##       education     < 22.5         to the left,  improve=262.3747, (0 missing)
##       BaP           < 0.053328     to the right, improve=232.7043, (0 missing)
##   Surrogate splits:
##       BaP           < 0.053328     to the right, agree=0.878, adj=0.485, (0 split)
##       pm2_5         < 4.810361     to the right, agree=0.827, adj=0.271, (0 split)
##       ap_pc2        < 0.8746175    to the left,  agree=0.792, adj=0.124, (0 split)
##       so2           < 0.3302972    to the left,  agree=0.781, adj=0.078, (0 split)
##       age_education splits as  ----LLLLLL-LLLLLLLRLR-LRRLRRRRRR-RRRRLLLLLR-LRLRLLRRLL-LLRLLR-LLR-RRLLLLL-----RR-----R-L, agree=0.779, adj=0.071, (0 split)
## 
## Node number 14: 11607 observations,    complexity param=0.008270266
##   predicted class=0  expected loss=0.3804601  P(node) =0.1545827
##     class counts:  7191  4416
##    probabilities: 0.620 0.380 
##   left son=28 (7462 obs) right son=29 (4145 obs)
##   Primary splits:
##       age_education                    splits as  ----LLLLLL-LRRRRRRRRR-RRLRRLRRLL-RRRRLRLLRR-RLRLLLRLRL-RR-RR--RRL-L-LLRRR------------L-R, improve=123.71070, (0 missing)
##       year                             splits as  R-LR, improve=107.79460, (0 missing)
##       education                        < 20.5         to the left,  improve= 90.28724, (0 missing)
##       occupation_of_respondent         splits as  LRRLRRRRRLRLLLRLLL, improve= 84.62865, (0 missing)
##       respondent_occupation_scale_c_14 splits as  LRLLLRRL, improve= 68.88653, (0 missing)
##   Surrogate splits:
##       education                        < 20.5         to the left,  agree=0.950, adj=0.861, (0 split)
##       occupation_of_respondent         splits as  LLLLRLLRRLRLLLRLLL, agree=0.738, adj=0.267, (0 split)
##       respondent_occupation_scale_c_14 splits as  LRLLLLRL, agree=0.733, adj=0.251, (0 split)
##       is_student                       < 0.5          to the left,  agree=0.709, adj=0.186, (0 split)
##       age_exact                        < 23.5         to the right, agree=0.676, adj=0.094, (0 split)
## 
## Node number 15: 3619 observations
##   predicted class=1  expected loss=0.3722023  P(node) =0.04819807
##     class counts:  1347  2272
##    probabilities: 0.372 0.628 
## 
## Node number 28: 7462 observations
##   predicted class=0  expected loss=0.326052  P(node) =0.09937938
##     class counts:  5029  2433
##    probabilities: 0.674 0.326 
## 
## Node number 29: 4145 observations,    complexity param=0.008270266
##   predicted class=0  expected loss=0.4784077  P(node) =0.05520337
##     class counts:  2162  1983
##    probabilities: 0.522 0.478 
##   left son=58 (2573 obs) right son=59 (1572 obs)
##   Primary splits:
##       year                     splits as  L-LR, improve=40.13885, (0 missing)
##       occupation_of_respondent splits as  LRLLRRRRRLRLLLRLLL, improve=18.33254, (0 missing)
##       marital_status           splits as  LRRRLRRRLRRLRLLRRRRRRLRLRLLRR, improve=17.86888, (0 missing)
##       type_of_community        splits as  LRLRL, improve=17.55254, (0 missing)
##       age_education            splits as  ------------LLRRRRRRR-RR-RL-RR---LRRR-R--LR-R-R---R-R--RR-RR--RR------RRR--------------R, improve=14.66121, (0 missing)
##   Surrogate splits:
##       type_of_community splits as  LLLRL, agree=0.777, adj=0.412, (0 split)
##       marital_status    splits as  RRLLLLLRLLLLLLLRRRLLLLLLRLRLL, agree=0.680, adj=0.155, (0 split)
##       isocntry          splits as  LL---LL---L-R----------LL------, agree=0.669, adj=0.127, (0 split)
##       country_code      splits as  LL---L---L-R--------LL------, agree=0.669, adj=0.127, (0 split)
##       o3                < 83.06345     to the right, agree=0.650, adj=0.076, (0 split)
## 
## Node number 58: 2573 observations
##   predicted class=0  expected loss=0.4240187  P(node) =0.03426737
##     class counts:  1482  1091
##    probabilities: 0.576 0.424 
## 
## Node number 59: 1572 observations
##   predicted class=1  expected loss=0.43257  P(node) =0.02093599
##     class counts:   680   892
##    probabilities: 0.433 0.567

# plot tree
plot(fit, uniform=TRUE,
   main="Classification Tree: Climate Change Is The Most Serious Threat")
text(fit, use.n=TRUE, all=TRUE, cex=.8)

## Warning in labels.rpart(x, minlength = minlength): more than 52 levels in a
## predicting factor, truncated for printout

saveRDS ( climate_awareness_air , file.path(tempdir(), "climate_panel_recoded.rds"), version = 2)

# not evaluated
saveRDS( climate_awareness_air, file = file.path("data-raw", "climate-panel_recoded.rds"))

Retrospective Survey Harmonization Case Study - Climate Awareness Change in Europe 2013-2019.

Fri, 05 Mar 2021 00:00:00 +0000

Retrospective survey harmonization comes with many challenges, as we have shown in the introduction to this tutorial case study. In this example, we will work with Eurobarometer’s data.

Please use the development version of retroharmonize:

devtools::install_github("antaldaniel/retroharmonize")

library(retroharmonize)
library(dplyr)       # this is necessary for the example 
library(lubridate)   # easier date conversion

## Warning: package 'lubridate' was built under R version 4.0.4

library(stringr)     # You can also use base R string processing functions

Get the Data

retroharmonize is not associated with Eurobarometer, or its creators, Kantar, or its archivists, GESIS. We assume that you have acquired the necessary files from GESIS after carefully reading their terms and you placed it on a path that you call gesis_dir. The precise documentation of the data we use can be found in this supporting blogpost. To reproduce this blogpost, you will need ZA5877_v2-0-0.sav, ZA6595_v3-0-0.sav, ZA6861_v1-2-0.sav, ZA7488_v1-0-0.sav, ZA7572_v1-0-0.sav in a directory that you will name gesis_dir.

#Not run in the blogpost. In the repo we have a saved version.
climate_change_files <- c("ZA5877_v2-0-0.sav", "ZA6595_v3-0-0.sav",  "ZA6861_v1-2-0.sav", 
                          "ZA7488_v1-0-0.sav", "ZA7572_v1-0-0.sav")

eb_waves <- read_surveys(file.path(gesis_dir, climate_change_files), .f='read_spss')

if (dir.exists("data-raw")) {
  save ( eb_waves,  file = file.path("data-raw", "eb_climate_change_waves.rda") )
}

if ( file.exists( file.path("data-raw", "eb_climate_change_waves.rda") )) {
  load (file.path( "data-raw", "eb_climate_change_waves.rda" ) )
} else {
  load (file.path("..", "..",  "data-raw", "eb_climate_change_waves.rda") )
}

The eb_waves nested list contains five surveys imported from SPSS to the survey class of retroharmonize. The survey class is a data.frame that retains important metadata for further harmonization.

document_waves (eb_waves)

## # A tibble: 5 x 5
##   id            filename           ncol  nrow object_size
##   <chr>         <chr>             <int> <int>       <dbl>
## 1 ZA5877_v2-0-0 ZA5877_v2-0-0.sav   604 27919   139352456
## 2 ZA6595_v3-0-0 ZA6595_v3-0-0.sav   519 27718   119370440
## 3 ZA6861_v1-2-0 ZA6861_v1-2-0.sav   657 27901   151397528
## 4 ZA7488_v1-0-0 ZA7488_v1-0-0.sav   752 27339   169465928
## 5 ZA7572_v1-0-0 ZA7572_v1-0-0.sav   348 27655    80562432

Beware the object sizes. If you work with many surveys, memory-efficient programming becomes imperative. We will be subsetting whenever possible.

Metadata analysis

As noted before, prepare to work with nested lists. Each imported survey is nested as a data frame in the eb_waves list.

Metadata: Protocol Variables

Eurobarometer calls certain metadata elements, like interviewee cooperation level or the date of a survey interview as protocol variable. Let’s start here. This will be our template to harmonize more and more aspects of the five surveys (which are, in fact, already harmonization of about 30 surveys conducted in a single ‘wave’ in multiple countries.)

# select variables of interest from the metadata
eb_protocol_metadata <- eb_climate_metadata %>%
  filter ( .data$label_orig %in% c("date of interview") |
             .data$var_name_orig == "rowid")  %>%
  suggest_var_names( survey_program = "eurobarometer" )

# subset and harmonize these variables in all nested list items of 'waves' of surveys
interview_dates <- harmonize_var_names(eb_waves, 
                                       eb_protocol_metadata )

# apply similar data processing rules to same variables
interview_dates <- lapply (interview_dates, 
                      function (x) x %>% mutate ( date_of_interview = as_character(.data$date_of_interview) )
                      )

# join the individual survey tables into a single table 
interview_dates <- as_tibble ( Reduce (rbind, interview_dates) )

# Check the variable classes.

vapply(interview_dates, function(x) class(x)[1], character(1))

##             rowid date_of_interview 
##       "character"       "character"

This is our sample workflow for each block of variables.

Get a unique identifier.
Add other variables
Harmonize the variable names
Subset the data leaving out anything that you do not harmonize in this block.
Apply some normalization in a nested list.
When the variables are harmonized to same name, class, merge them into a data.frame-like tibble object.

Now finish the harmonization. Wednesday, 31st October 2018 should become a Date type 2018-10-31.

require(lubridate)
harmonize_date <- function(x) {
  x <- tolower(as.character(x))
  x <- gsub("monday|tuesday|wednesday|thursday|friday|saturday|sunday|\\,|th|nd|rd|st", "", x)
  x <- gsub("decemberber", "december", x) # all those annoying real-life data problems!
  x <- stringr::str_trim (x, "both")
  x <- gsub("^0", "", x )
  x <- gsub("\\s\\s", "\\s", x)
  lubridate::dmy(x) 
}

interview_dates <- interview_dates %>%
  mutate ( date_of_interview = harmonize_date(.data$date_of_interview) )

vapply(interview_dates, function(x) class(x)[1], character(1))

##             rowid date_of_interview 
##       "character"            "Date"

To avoid duplication of row IDs in surveys that may not be unique in different surveys, we created a simple, sequential ID for each survey, including the ID of the original file.

set.seed(2021)
sample_n(interview_dates, 6)

## # A tibble: 6 x 2
##   rowid               date_of_interview
##   <chr>               <date>           
## 1 ZA7488_v1-0-0_7016  2018-10-28       
## 2 ZA7488_v1-0-0_19187 2018-11-02       
## 3 ZA6861_v1-2-0_1218  2017-03-18       
## 4 ZA6861_v1-2-0_4142  2017-03-21       
## 5 ZA7572_v1-0-0_12363 2019-04-17       
## 6 ZA7572_v1-0-0_8071  2019-04-18

After this type-conversion problem let’s see an issue when an original SPSS variable can have two meaningful R representations.

Metadata: Geographical information

Let’s continue with harmonizing geographical information in the files. In this example, var_name_suggested will contain the harmonized variable name. It is likely that you have to make this call, after carefully reading the original questionnaires and codebooks.

eb_regional_metadata <- eb_climate_metadata %>%
  filter ( grepl( "rowid|isocntry|^nuts$", .data$var_name_orig)) %>%
  suggest_var_names( survey_program = "eurobarometer" ) %>%
  mutate ( var_name_suggested = case_when ( 
    var_name_suggested == "region_nuts_codes"     ~ "geo",
    TRUE ~ var_name_suggested ))

The harmonize_var_names() takes all variables in the subsetted, geographical metadata table, and brings them to the harmonized var_name_suggested name. The function subsets the surveys to avoid the presence of non-harmonized variables. All regional NUTS codes become geo in our case:

geography <- harmonize_var_names(eb_waves, 
                                 eb_regional_metadata)

If you are used to work with single survey files, you are likely to work in a tabular format, which easily converts into a data.frame like object, in our example, to tidyverse’s tibble. However, when working with longitudinal data, it is far simpler to work with nested lists, because the tables usually have different dimensions (neither the rows corresponding to observations or the columns are the same across all survey files.)

In the nested list, each list element is a single, tabular-format survey. (In fact, the survey are in retroharmonize’s survey class, which is a rich tibble that contains the metadata and the processing history of the survey.)

The regional information in the Eurobarometer files is contained in the nuts variable. We want to keep both the original labels and values. The original values are the region’s codes, and the labels are the names. The easiest and fastest solution is the base R lapply loop.

geography <- lapply ( geography, 
                      function (x) x %>% mutate ( region = as_character(geo), 
                                                  geo    = as.character(geo) )  
)

Because each table has exactly the same columns, we can simply use rbind() and reduce the list to a modern data.frame, i.e. a tibble.

geography <- as_tibble ( Reduce (rbind, geography) )

Let’s see a dozen cases:

set.seed(2021)
sample_n(geography, 12)

## # A tibble: 12 x 4
##    rowid               isocntry geo   region              
##    <chr>               <chr>    <chr> <chr>               
##  1 ZA7488_v1-0-0_7016  SI       SI012 Podravska           
##  2 ZA7488_v1-0-0_19187 PL       PL63  Pomorskie           
##  3 ZA6861_v1-2-0_1218  DK       DK02  Sjaelland           
##  4 ZA6861_v1-2-0_4142  FI       FI1B  Helsinki-Uusimaa    
##  5 ZA7572_v1-0-0_12363 SE       SE12  Oestra Mellansverige
##  6 ZA7572_v1-0-0_8071  IT       ITH   Nord-Est [IT]       
##  7 ZA6861_v1-2-0_6145  IE       IE021 Dublin              
##  8 ZA6861_v1-2-0_24638 RO       RO31  South [RO]          
##  9 ZA7488_v1-0-0_11315 CY       CY    REPUBLIC OF CYPRUS  
## 10 ZA6595_v3-0-0_27568 HR       HR041 Grad Zagreb         
## 11 ZA7572_v1-0-0_17397 CZ       CZ06  Jihovychod          
## 12 ZA6861_v1-2-0_10993 PT       PT17  Lisboa

The idea is that we do similar variable harmonization block by block, and eventually we will join them together. Next step: socio-demography and weights.

Socio-demography and Weights

There are a few peculiar issues to look out for. This example shows that survey harmonization requires plenty of expert judgment, and you cannot fully automate the process.

The Eurobarometer archives do not use all weight and demographic variable names consistently. For example, the wex variable, which is a projected weight for the country’s 15 years old or older population is sometimes called wex, sometimes wextra. The individual survey’s post-stratification weight is the w1 variable, but this is not necessarily what you need to use.

The suggest_var_names() function has a parameter for survey_program = "eurobaromater" which normalizes a bit the most used variables. For example, all variations of wex, wextra wil be noramlized to wex. You can ignore this parameter and use your own names, too.

eb_demography_metadata  <- eb_climate_metadata %>%
  filter ( grepl( "rowid|isocntry|^d8$|^d7$|^wex|^w1$|d25|^d15a|^d11$", .data$var_name_orig) ) %>%
  suggest_var_names( survey_program = "eurobarometer")

As you can see, using the original labels would not help, because they also contain various alterations.

eb_demography_metadata %>%
  select ( filename, var_name_orig, label_orig, var_name_suggested ) %>%
  filter (var_name_orig %in% c("wex", "wextra") )

##            filename var_name_orig                                  label_orig
## 1 ZA5877_v2-0-0.sav        wextra      weight extrapolated population 15 plus
## 2 ZA6595_v3-0-0.sav        wextra      weight extrapolated population 15 plus
## 3 ZA6861_v1-2-0.sav           wex weight extrapolated population aged 15 plus
## 4 ZA7488_v1-0-0.sav           wex weight extrapolated population aged 15 plus
## 5 ZA7572_v1-0-0.sav           wex weight extrapolated population aged 15 plus
##   var_name_suggested
## 1                wex
## 2                wex
## 3                wex
## 4                wex
## 5                wex

demography <- harmonize_var_names ( waves = eb_waves, 
                                    metadata = eb_demography_metadata )

Socio-demographic variables like level of highest education or occupation are rather country-specific. Eurobarometer uses standardized occupation and marital status scales, and a proxy for education levels, age of leaving full-time education.

This is a particularly tricky variable, because it’s coding in fact contains three different variables - school leaving age, except for students, and except for people who did not finish their compulsory primary school. And while school leaving age was a good proxy since the 1970s, in the age when the EU is promoting life-long-learning becomes less and less useful, as people stop and re-start their education throughout their lives.

example <- demography[[1]] %>%
  mutate ( across ( -any_of(c("rowid", "w1", "wex")), as_character) ) %>%
  mutate ( across (any_of(c("w1", "wex")), as_numeric) )
unique ( example$age_education )

##  [1] "22"                     "25"                     "17"                    
##  [4] "19"                     "12"                     "23"                    
##  [7] "18"                     "20"                     "21"                    
## [10] "14"                     "24"                     "16"                    
## [13] "26"                     "15"                     "Still studying"        
## [16] "DK"                     "31"                     "29"                    
## [19] "27"                     "13"                     "32"                    
## [22] "28"                     "30"                     "53"                    
## [25] "42"                     "62"                     "40"                    
## [28] "No full-time education" "Refusal"                "37"                    
## [31] "39"                     "34"                     "35"                    
## [34] "47"                     "36"                     "45"                    
## [37] "51"                     "33"                     "43"                    
## [40] "38"                     "49"                     "46"                    
## [43] "41"                     "57"                     "7"                     
## [46] "48"                     "44"                     "50"                    
## [49] "56"                     "8"                      "11"                    
## [52] "10"                     "9"                      "75 years"              
## [55] "6"                      "3"                      "54"                    
## [58] "55"                     "60"                     "64"                    
## [61] "2 years"                "58"                     "52"                    
## [64] "72"                     "61"                     "4"                     
## [67] "63"

The seamingly trival age_exact variable has its own issues, too:

unique ( example$age_exact)

##  [1] "54"       "66"       "56"       "53"       "33"       "72"      
##  [7] "83"       "62"       "86"       "77"       "64"       "46"      
## [13] "44"       "59"       "60"       "67"       "63"       "20"      
## [19] "43"       "37"       "78"       "49"       "90"       "45"      
## [25] "28"       "29"       "30"       "39"       "51"       "38"      
## [31] "41"       "71"       "25"       "48"       "79"       "88"      
## [37] "61"       "85"       "70"       "35"       "81"       "52"      
## [43] "57"       "27"       "47"       "15 years" "21"       "42"      
## [49] "32"       "68"       "36"       "34"       "19"       "31"      
## [55] "26"       "23"       "24"       "22"       "16"       "84"      
## [61] "65"       "18"       "55"       "40"       "50"       "73"      
## [67] "69"       "87"       "89"       "74"       "75"       "98 years"
## [73] "76"       "80"       "58"       "82"       "17"       "93"      
## [79] "91"       "92"       "95"       "94"       "97"

Let’s see all the strange labels attached to age-type variables:

collect_val_labels(metadata = eb_demography_metadata %>%
                     filter ( var_name_suggested %in% c("age_exact", "age_education")) )

##  [1] "2 years"                  "75 years"                
##  [3] "No full-time education"   "Still studying"          
##  [5] "15 years"                 "98 years"                
##  [7] "96 years"                 "[NOT CLEARLY DOCUMENTED]"
##  [9] "74 years"                 "99 and older"            
## [11] "Refusal"                  "87 years"                
## [13] "DK"                       "88 years"

We must handle many exception, so we created a function for this purpose:

remove_years  <- function(x) { 
  x <- gsub("years|and\\solder", "", tolower(x))
  stringr::str_trim (x, "both")}

process_demography <- function (x) { 
  
  x %>% mutate ( across ( -any_of(c("rowid", "w1", "wex")), as_character) ) %>%
    mutate ( across (any_of(c("w1", "wex")), as_numeric) ) %>%
    mutate ( across (contains("age"), remove_years)) %>%
    mutate ( age_exact = as.numeric (age_exact)) %>%
    mutate ( is_student = ifelse ( tolower(age_education) == "still studying", 
                                   1, 0), 
             no_education = ifelse ( tolower(age_education) == "no full-time education", 1, 0)) %>%
    mutate ( education = case_when (
      grepl("studying", age_education) ~ age_exact, 
      grepl ("education", age_education)  ~ 14, 
      grepl ("refus|document|dk", tolower(age_education)) ~ NA_real_,
      TRUE ~ as.numeric(age_education)
    ))  %>%
    mutate ( education = case_when ( 
      education < 14 ~ NA_real_, 
      education > 30 ~ 30, 
      TRUE ~ education )) 
}

demography <- lapply ( demography, process_demography )

## Warning in eval_tidy(pair$rhs, env = default_env): NAs introduced by coercion

## Warning in mask$eval_all_mutate(quo): NAs introduced by coercion

## Warning in eval_tidy(pair$rhs, env = default_env): NAs introduced by coercion

## Warning in eval_tidy(pair$rhs, env = default_env): NAs introduced by coercion

## Warning in eval_tidy(pair$rhs, env = default_env): NAs introduced by coercion

## Warning in eval_tidy(pair$rhs, env = default_env): NAs introduced by coercion

## WE'll full join and not use rbind, because we have different variables in different waves.
demography <- Reduce ( full_join, demography )

## Joining, by = c("rowid", "isocntry", "w1", "wex", "marital_status", "age_education", "age_exact", "occupation_of_respondent", "occupation_of_respondent_recoded", "respondent_occupation_scale_c_14", "type_of_community", "is_student", "no_education", "education")
## Joining, by = c("rowid", "isocntry", "w1", "wex", "marital_status", "age_education", "age_exact", "occupation_of_respondent", "occupation_of_respondent_recoded", "respondent_occupation_scale_c_14", "type_of_community", "is_student", "no_education", "education")
## Joining, by = c("rowid", "isocntry", "w1", "wex", "marital_status", "age_education", "age_exact", "occupation_of_respondent", "occupation_of_respondent_recoded", "respondent_occupation_scale_c_14", "type_of_community", "is_student", "no_education", "education")
## Joining, by = c("rowid", "isocntry", "w1", "wex", "marital_status", "age_education", "age_exact", "occupation_of_respondent", "occupation_of_respondent_recoded", "respondent_occupation_scale_c_14", "type_of_community", "is_student", "no_education", "education")

Now let’s see what we have here:

set.seed(2021)
sample_n(demography, 12)

## # A tibble: 12 x 14
##    rowid    isocntry    w1    wex marital_status        age_education  age_exact
##    <chr>    <chr>    <dbl>  <dbl> <chr>                 <chr>              <dbl>
##  1 ZA7488_~ SI       0.828  1428. (Re-)Married: withou~ 19                    43
##  2 ZA7488_~ PL       1.01  32830. (Re-)Married: withou~ 19                    64
##  3 ZA6861_~ DK       0.641  3100. (Re-)Married: withou~ 22                    78
##  4 ZA6861_~ FI       1.83   8601. (Re-)Married: childr~ 30                    38
##  5 ZA7572_~ SE       0.342  2645. (Re-)Married: withou~ 17                    68
##  6 ZA7572_~ IT       0.630 32287. (Re-)Married: childr~ 20                    40
##  7 ZA6861_~ IE       0.868  3054. (Re-)Married: childr~ 32                    42
##  8 ZA6861_~ RO       0.724 11805. (Re-)Married: withou~ 14                    59
##  9 ZA7488_~ CY       0.691  1013. (Re-)Married: childr~ 18                    67
## 10 ZA6595_~ HR       0.580  2098. Single living w part~ 27                    30
## 11 ZA7572_~ CZ       1.86  16908. Single: without chil~ still studying        20
## 12 ZA6861_~ PT       0.932  7448. Widow: with children  no full-time ~        84
## # ... with 7 more variables: occupation_of_respondent <chr>,
## #   occupation_of_respondent_recoded <chr>,
## #   respondent_occupation_scale_c_14 <chr>, type_of_community <chr>,
## #   is_student <dbl>, no_education <dbl>, education <dbl>

Harmonizing Variable Labels

So far we have been working with metadata, weights and socio-demography. In other words, we have not even started the desired harmonization of climate change awareness. The methodology is the same, but here we really must look out for the answer options in the questionnaire. (Refer to our data summary again here.)

climate_awareness_metadata <- eb_climate_metadata %>%
  suggest_var_names( survey_program = "eurobarometer" ) %>%
  filter ( .data$var_name_suggested  %in% c("rowid",
                                            "serious_world_problems_first", 
                                             "serious_world_problems_climate_change")
  ) 

hw <- harmonize_var_names ( waves = eb_waves, 
                            metadata = climate_awareness_metadata )

The retroharmoinze package comes with a generic harmonize_values() function that will change the value labels of categorical variables (including binary ones) to a unitary format. It will also take care of various types of missing values.

First, let’s go back to our metadata and collect all value labels that will show up with collect_val_labels():

collect_val_labels(climate_awareness_metadata)

##  [1] "Climate change"                            
##  [2] "International terrorism"                   
##  [3] "Poverty, hunger and lack of drinking water"
##  [4] "Spread of infectious diseases"             
##  [5] "The economic situation"                    
##  [6] "Proliferation of nuclear weapons"          
##  [7] "Armed conflicts"                           
##  [8] "The increasing global population"          
##  [9] "Other (SPONTANEOUS)"                       
## [10] "None (SPONTANEOUS)"                        
## [11] "Not mentioned"                             
## [12] "Mentioned"                                 
## [13] "DK"

In this case, we want to select Climate change as the mentioned most serious problem, and Climate change taken from a list of three serious problems. The first question type is a single-choice one, where Climate change is either mentioned, or the alternative answer is labeled as Not mentioned. In the multiple choice case, the alternative may be something else, for example, Spread of infectious diseases, as we all well know by 2021.

We want to see who thought Climate change was the most serious problem, or one of the most serious problems, so we label each mentions of Climate change as mentioned and we pair it with a numeric value of 1. All other cases are labeled as not_mentioned, with the exceptions of various missing observations, which in these cases are Do not know answers, Declined to answer cases, and Inappropriate cases [The latter one is Eurobarometer’s label for questions that were for one reason or other not asked from a particular interviewee – for example, because the Turkish Cypriot community received a different questionnaire.]

# positive cases
label_1 = c("^Climate\\schange", "^Mentioned")
# missing cases 
na_labels <- collect_na_labels( climate_awareness_metadata)
na_labels

## [1] "DK"                             "Inap. (10 or 11 in qa1a)"      
## [3] "Inap. (coded 10 or 11 in qc1a)" "Inap. (coded 10 or 11 in qb1a)"

# negative cases
label_0 <- collect_val_labels( climate_awareness_metadata)
label_0 <- label_0[! label_0 %in% label_1 ]

The harmonize_serious_problems() function harmonizes the labels within the special labeled class of retroharmonize. This class retains all information to give categorical variables a character or numeric representation, and various processing metadata for documentation purposes. While this class is very reach (it contains whatever was imported from SPSS’s proprietary data format and the history), it is not suitable for statistical analysis. We could, of course, directly call the harmonize_values() from the retroharmonize package, but the parameterization would be very complicated even in a simple function call, not to mention a looped call. Because this function is the heart of the retroharmonize package, it has a tutorial article on its own.

harmonize_serious_problems <- function(x) {
  label_list <- list(
    from = c(label_0, label_1, na_labels), 
    to = c( rep ( "not_mentioned", length(label_0) ),   # use the same order as in from!
            rep ( "mentioned", length(label_1) ),
            "do_not_know", "inap", "inap", "inap"), 
    numeric_values = c(rep ( 0, length(label_0) ), # use the same order as in from!
                       rep ( 1, length(label_1) ),
                       99997,99999,99999,99999)
  )
  
  harmonize_values(x, 
                   harmonize_labels = label_list, 
                   na_values = c("do_not_know"=99997,
                                 "declined"=99998,
                                 "inap"=99999), 
                   remove = "\\(|\\)|\\[|\\]|\\%"
  )
}

Our objects are rather big in memory, so first, let’s remove the surveys that do not contain these world problem variables. In this cases, the subsetted and harmonized surveys in the nested list have only one columns, i.e. the rowid.

hw <- hw[unlist ( lapply ( hw, ncol)) > 1 ]

Now we have a smaller problem to deal with. With many surveys, it is easy to fill up your computer’s memory, so let’s start building up our joined panel data from a smaller set of nested, subsetted surveys.

hw <- lapply ( hw, function (x) x %>% mutate ( across ( contains("problem"), harmonize_serious_problems) ) )

Our lapply loop calls an anonymous function which in turn calls the harmonize_serious_problems parameterized version of the harmonize_values() on all variables that have problem in their names.

once we are done, our variables have harmonized names, and harmonized values, and harmonized label, but they are stored in the complex retroharmonize_labelled_spss_survey class, inherited from the haven_labelled_spss in haven.

We reduced our single and multiple choice questions to binary choice variables. We can now give them a numeric representation. Be mindful that retroharmonize has special methods for its special labeled class that retains metadata from SPSS. This means that as_character and as_numeric knows how to handle various types of missing values, whereas the base R as.character and as.numeric may coerce special values to unwanted results. This is particularly dangerous with numeric variables – and this is the reason why we introduced a new set of S3 objects and methods in the package.

We will ignore the differences between various forms of missingness, i.e. the person said that she did not know, or did not want to answer, or for some reason was not asked in the survey. In a more descriptive, non-harmonized analysis you would probably want to explore them as various ‘categories’ and use a character representation.

hw <- lapply ( hw, function(x) x %>% mutate ( across ( contains("problem"), as_numeric) ))

hw <- Reduce ( full_join, hw) # we must use joins instead of binds because the number of columns vary.

Let’s see what we have:

set.seed(2021)
sample_n (hw, 12)

## # A tibble: 12 x 3
##    rowid             serious_world_problems_fi~ serious_world_problems_climate_~
##    <chr>                                  <dbl>                            <dbl>
##  1 ZA6595_v3-0-0_23~                          0                               NA
##  2 ZA7572_v1-0-0_70~                          0                                0
##  3 ZA6595_v3-0-0_18~                          0                               NA
##  4 ZA6861_v1-2-0_27~                          0                                0
##  5 ZA6595_v3-0-0_26~                          0                               NA
##  6 ZA7572_v1-0-0_19~                          0                                1
##  7 ZA5877_v2-0-0_16~                          0                                0
##  8 ZA6861_v1-2-0_12~                          0                                0
##  9 ZA7572_v1-0-0_17~                          0                                0
## 10 ZA5877_v2-0-0_17~                          0                                1
## 11 ZA6861_v1-2-0_41~                          0                                0
## 12 ZA6861_v1-2-0_61~                          0                                1

Creating the Longitudional Table

Now we just need to join the partial table by the rowid together:

#start from the smallest (we removed the survey that had no relevant questionnaire item)
panel <- hw %>%
  left_join ( geography, by = 'rowid' ) 

panel <- panel %>%
  left_join ( demography, by = c("rowid", "isocntry") ) 

panel <- panel %>%
  left_join ( interview_dates, by = 'rowid' )

And let’s see a small sample:

sample_n(panel, 12)

## # A tibble: 12 x 19
##    rowid  serious_world_pr~ serious_world_pr~ isocntry geo   region    w1    wex
##    <chr>              <dbl>             <dbl> <chr>    <chr> <chr>  <dbl>  <dbl>
##  1 ZA686~                 0                 0 ES       ES41  Casti~ 1.21  46787.
##  2 ZA686~                 0                 0 RO       RO31  South~ 0.724 11805.
##  3 ZA686~                 0                 0 SK       SK02  Zapad~ 0.774  3499.
##  4 ZA757~                 0                 1 PT       PT16  Centr~ 1.11   9336.
##  5 ZA659~                 1                NA HR       HR041 Grad ~ 0.580  2098.
##  6 ZA659~                 1                NA RO       RO21  North~ 1.21  20160.
##  7 ZA686~                 0                 0 PT       PT17  Lisboa 0.932  7448.
##  8 ZA659~                 0                NA GB-GBN   UKI   London 0.994 50133.
##  9 ZA757~                 0                 0 CY       CY    REPUB~ 0.594   874.
## 10 ZA686~                 0                 0 LT       LT003 Klaip~ 0.623  1564.
## 11 ZA757~                 0                 0 IE       IE013 West ~ 0.490  1651.
## 12 ZA659~                 0                NA LT       LT003 Klaip~ 1.16   2917.
## # ... with 11 more variables: marital_status <chr>, age_education <chr>,
## #   age_exact <dbl>, occupation_of_respondent <chr>,
## #   occupation_of_respondent_recoded <chr>,
## #   respondent_occupation_scale_c_14 <chr>, type_of_community <chr>,
## #   is_student <dbl>, no_education <dbl>, education <dbl>,
## #   date_of_interview <date>

saveRDS ( panel, file.path(tempdir(), "climate_panel.rds"), version = 2)

# not evaluated
saveRDS( panel, file = file.path("data-raw", "climate-panel.rds"), version=2)

Putting It on a Map

This is not the end of the story. If you put all this on a map, the results are a bit disappointing.

Why? Because sub-national (provincial, state, county, district, parish) borders are changing all the time - within the EU and everywhere. The next step is to harmonize the geographical information. We have another CRAN released package to help you with. See the next post: Regional Climate Change Awareness Dataset.

Our Music Observatory in the Jump European Music Market Accelerator: Meet the 2021 Fellows and their Tutors

Thu, 04 Mar 2021 15:00:00 +0200

According to the announcement of JUMP, the European Music Market Accelerator, after a careful screening of all applications received, the selection committee composed of all JUMP board members has selected the most promising ideas and projects to be developed together with renowned tutors for this 2021 fellowship.

For nine months, the 20 fellows living in many European countries will develop their innovative projects, while receiving a comprehensive 360° training. In addition to specialised workshops by highly qualified experts, each fellow will receive one-on-one tutoring sessions from the most renowned music professionals coming from all over Europe.

The 20 selected projects cover a great variety of urgent needs faced within the music sector. They will:

help fostering social change with projects focusing on diversity in the industry, more fairness and transparency as well as raising awareness on timely issues.
enhance technological development with projects using blockchain, immersive sound and VR and AR.
build bridges between different key actors of the ecosystem.

Download the entire JUMP press release.

Reprex’s project, the automated Demo Music Observatory will be represented by Daniel Antal, co-founder of Reprex among other building bridges projects. This project offers a different approach to the planned European Music Observatory based on the principles of open collaboration, which allows contributions from small organizations and even individuals, and which provides higher levels of quality in terms of auditability, timeliness, transparency and general ease of use. Our open collaboration approach allows to power trustworthy, ethical AI systems like our Listen Local that we started out from Slovakia with the support of the Slovak Arts Council.

JUMP fellows building bridges between different key actors of the ecosystem.

Apart from our Demo Music Observatory the build bridges section Groovly with Martin Zenzerovich, From Play To Rec by Jeremy Dunne, Hajde Radio by Thibaut Boudaud, LowDee by Alex Davidson and ONO-HU! by Gina Akers.

Meet all the JUMP 2021 Fellows, including the technology and social change professionals!

Reprex is a start-up company based in the Netherlands and the United States that validated its early products in the Yes!Delft AI+Blockchain Lab in the Hague. In 2021 we joined the Dutch AI Coalition – NL AIC and requested membership in the European AI Alliance. Reprex is committed to applying reproducible in an open collaboration with our business, scientific, policy and civil society partners, and facilitate the use of open data and open-source software. Many fellows in the program are connected to other regions, like North America and Australia – because music is one of the most globalized industries and forms of art in the world! Reprex is a startup based in the Netherlands and the United States, and we are very excited to collaborate with our peers in new European territories, and in Canada and Australia.

Hope to meet you in these great events - maybe not only online!

Further links:

From Play to Rec on Facebook
HAJDE FR/EN

What is Retrospective Survey Harmonization?

Thu, 04 Mar 2021 00:00:00 +0000

Reproducible ex post harmonization of survey microdata

Retrospective survey harmonization allows the comparison of opinion poll data conducted in different countries or time. In this example we are working with data from surveys that were ex ante harmonized to a certain degree – in our tutorials we are choosing questions that were asked in the same way in many natural languages. For example, you can compare what percentage of the European people in various countries, provinces and regions thought climate change was a serious world problem back in 2013, 2015, 2017 and 2019.

We developed the retroharmonize R package to help this process. We have tested the package with about 80 Eurobarometer, 5 Afrobarometer survey files extensively, and a bit with Arabbarometer files. This allows the comparison of various survey answers in about 70 countries. This policy-oriented survey programs were designed to be harmonized to a certain degree, but their ex post harmonization is still necessary, challenging and errorprone. Retrospective harmonization includes harmonization of the different coding used for questions and answer options, post-stratification weights, and using different file formats.

Eurobarometer, Afrobaromer, Arab Barometer and Latinobarómetro make survey files that are harmonized across countries available for research with various terms. Our retroharmonize is not affiliated with them, and to run our examples, you must visit their websites, carefully read their terms, agree to them, and download their data yourself. What we add as a value is that we help to connect their files across time (from different years) or across these programs.

The survey programs mentioned above publish their data in the proprietary SPSS format. This file format can be imported and translated to R objects with the haven package; however, we needed to re-design haven’s labelled_spss class to maintain far more metadata, which, in turn, a modification of the labelled class. The haven package was designed and tested with data stored in individual SPSS files.

The author of labelled, Joseph Larmarange describes two main approaches to work with labelled data, such as SPSS’s method to store categorical data in the Introduction to labelled.

Two main approaches of labelled data conversion.

Our approach is a further extension of Approach B. Survey harmonization in our case always means the joining data from several SPSS files, which requires a consistent coding among several data sources. This means that data cleaning and recoding must take place before conversion to factors, character or numeric vectors. This is particularly important with factor data (and their simple character conversions) and numeric data that occasionally contains labels, for example, to describe the reason why certain data is missing. Our tutorial vignette labelled_spss_survey gives you more information about this.

In the next series of tutorials, we will deal with an array of problems. These are not for the faint heart – you need to have a solid intermediate level of R to follow.

Tidy, joined survey data

The original files identifiers may not be unique, we have to create new, truly unique identifiers. Weighting may not be straightforward.
Neither the number of observations or the number of variables (which represents the survey questions and their translation to coded data) is the same. Certain data may be only present in one survey and not the other. This means that you will likely to run loops on lists and not data.frames, but eventually you must carefully join them.

Class conversion

Similar questions may be imported from a non-native R format, in our case, from an SPSS files, in an inconsistent manner. SPSS’s variable formats cannot be translated unambiguously to R classes. retroharmonize introduced a new S3 class system that handles this problem, but eventually you will have to choose if you want to see a numeric or character coding of each categorical variable.
The harmonized surveys, with harmonized variable names and harmonized value labels, must be brought to consistent R representations (most statistical functions will only work on numeric, factor or character data) and carefully joined into a single data table for analysis.

Harmonization of variables and variable labels

Same variables may come with dissimilar variable names and variable labels. It may be a challenge to match age with age. We need to harmonize the names of variables.
The harmonized variables may have different labeling. One may call refused answers as declined and the other refusal. On a simple choice, climate change may be ‘Climate change’ or Problem: Climate change. Binary choices may have survey-specific coding conventions. Value labels must be harmonized. There are good tools to do this in a single file - but we have to work with several of them.

Missing value harmonization

There are likely to be various types of missing values. Working with missing values is probably where most human judgment is needed. Why are some answers missing: was the question not asked in some questionnaires? Is there a coding error? Did the respondent refuse the question, or sad that she did not have an answer? retroharmonize has a special labeled vector type that retains this information from the raw data, if it is present, but you must make the judgment yourself – in R, eventually you will either create a missing category, or use NA_character_ or NA_real_.

That’s a lot to put on your plate.

It is unlikely that you will be able to work with completely unfamiliar survey programs if you do not have a strong intermediate level of R. Our package comes with tutorials for Eurobarometer, Afrobarometer and our development version already covers Arab Barometer, highlighting some peculiar issues with these survey programs, that we hope to give a head start for less experienced R users.

Eurobarometer Surveys Used In Our Project

Wed, 03 Mar 2021 00:00:00 +0000

In our tutorial series, we are going to harmonize the following questionnaire items from five Eurobarometer harmonized survey files. The Eurobarometer survey files are harmonized across countries, but they are only partially harmonized in time.

All data must be downloaded from the GESIS Data Archive in Cologne. We are not affiliated with GESIS and you must read and accept their terms to use the data.

Eurobarometer 80.2 (2013)

GESIS Data Archive, Cologne. ZA5877 Data file Version 2.0.0, https://doi.org/10.4232/1.12792

Data file: ZA6595 data file (European Commission 2017).
Questionnaire: Eurobarometer 83.4 Basic Bilingual Questionnaire
Citation: ZA6595 Bibtex

QA1a Which of the following do you consider to be the single most serious problem facing the world as a whole? (single choice)

QA1b Which others do you consider to be serious problems? (multiple choice)

QA2 And how serious a problem do you think climate change is at this moment? Please use a scale from 1 to 10, with '1' meaning it is "not at all a serious problem (scale 1-10)

QA4 To what extent do you agree or disagree with each of the following statements? - Fighting climate change and using energy more efficiently can boost the economy and jobs in the EU (agreement-disagreement 4-scale)

QA4 To what extent do you agree or disagree with each of the following statements? - Reducing fossil fuel imports from outside the EU could benefit the EU economically (agreement-disagreement 4-scale)

QA5 Have you personally taken any action to fight climate change over the past six months? (binary)

Eurobarometer 83.4 (2015)

European Commission, Brussels; Directorate General Communication COMM.A.1 ´Strategy, Corporate Communication Actions and Eurobarometer´GESIS Data Archive, Cologne. ZA6595 Data file Version 3.0.0, https://doi.org/10.4232/1.13146

Data file: ZA6595 data file (European Commission 2018).
Questionnaire: Eurobarometer 83.4 Basic Bilingual Questionnaire
Citation: ZA6595 Bibtex

Eurobarometer 87.1 (2017)

European Commission, Brussels; Directorate General Communication, COMM.A.1 ‘Strategic Communication’; European Parliament, Directorate-General for Communication, Public Opinion Monitoring Unit GESIS Data Archive, Cologne. ZA6861 Data file Version 1.2.0, https://doi.org/10.4232/1.12922

Data file: ZA6861 data file.
Questionnaire: Eurobarometer 90.2 Basic Bilingual Questionnaire
Citation: ZA6861 Bibtex

QC1a Which of the following do you consider to be the single most serious problem facing the world as a whole? (single choice)

QC1b Which others do you consider to be serious problems? (multiple choice)

QC2 And how serious a problem do you think climate change is at this moment? Please use a scale from 1 to 10, with '1' meaning it is "not at all a serious problem (scale 1-10)

Qc4 To what extent do you agree or disagree with each of the following statements? - Fighting climate change and using energy more efficiently can boost the economy and jobs in the EU (agreement-disagreement 4-scale)

Qc4 To what extent do you agree or disagree with each of the following statements? - Promoting EU expertise in new clean technologies to countries outside the EU can benefit the EU economically (agreement-disagreement 4-scale)

Qc4 To what extent do you agree or disagree with each of the following statements? - Reducing fossil fuel imports from outside the EU can benefit the EU economically (agreement-disagreement 4-scale)

Qc4 To what extent do you agree or disagree with each of the following statements? - Reducing fossil fuel imports from outside the EU can increase the security of EU energy supplies (agreement-disagreement 4-scale)

Qc4 To what extent do you agree or disagree with each of the following statements? - More public financial support should be given to the transition to clean energies even if it means subsidies to fossil fuels should be reduced. (agreement-disagreement 4-scale)

Qc5 Have you personally taken any action to fight climate change over the past six months? (binary)

Eurobarometer 90.2 (2018)

European Commission, Brussels; Directorate General Communication, COMM.A.3 ‘Media Monitoring and Eurobarometer’ GESIS Data Archive, Cologne. ZA7488 Data file Version 1.0.0, https://doi.org/10.4232/1.13289

Data file: ZA7488 data file (European Commission 2019a)
Questionnaire: Eurobarometer 90.2 Basic Bilingual Questionnaire
Citation: ZA7488 Bibtex

QB5 To what extent do you agree or disagree with each of the following statements? - Fighting climate change and using energy more efficiently can boost the economy and jobs in the EU (agreement-disagreement 4-scale)

QB5 To what extent do you agree or disagree with each of the following statements? - Promoting EU expertise in new clean technologies to countries outside the EU can benefit the EU economically (agreement-disagreement 4-scale)

QB5 To what extent do you agree or disagree with each of the following statements? - Reducing fossil fuel imports from outside the EU can benefit the EU economically (agreement-disagreement 4-scale)

QB5 To what extent do you agree or disagree with each of the following statements? - Reducing fossil fuel imports from outside the EU can increase the security of EU energy supplies (agreement-disagreement 4-scale)

QB5 To what extent do you agree or disagree with each of the following statements? - More public financial support should be given to the transition to clean energies even if it means subsidies to fossil fuels should be reduced. (agreement-disagreement 4-scale)

Eurobarometer 91.3 (2019)

European Commission, Brussels; Directorate General Communication, COMM.A.3 ‘Media Monitoring and Eurobarometer’ GESIS Data Archive, Cologne. ZA7572 Data file Version 1.0.0, https://doi.org/10.4232/1.13372

Data file: ZA7572 data file (European Commission 2019b).
Questionnaire: Eurobarometer 91.3 Basic Bilingual Questionnaire
Citation: ZA7572 Bibtex

QB4 To what extent do you agree or disagree with each of the following statements? - Taking action on climate change will lead to innovation that will make EU companies more competitive (N) (agreement-disagreement 4-scale)

QB4 To what extent do you agree or disagree with each of the following statements? - Promoting EU expertise in new clean technologies to countries outside the EU can benefit the EU economically (agreement-disagreement 4-scale)

QB4 To what extent do you agree or disagree with each of the following statements? - Reducing fossil fuel imports from outside the EU can benefit the EU economically (agreement-disagreement 4-scale)

QB4 To what extent do you agree or disagree with each of the following statements? - Adapting to the adverse impacts of climate change can have positive outcomes for citizens in the EU (agreement-disagreement 4-scale)

QB5 Have you personally taken any action to fight climate change over the past six months? (binary)

References

European Commission, Brussels. 2017. “Eurobarometer 80.2 (2013).” GESIS Data Archive, Cologne. ZA5877 Data file Version 2.0.0, https://doi.org/10.4232/1.12792. https://doi.org/10.4232/1.12792.

———. 2018. “Eurobarometer 83.4 (2015).” GESIS Data Archive, Cologne. ZA6595 Data file Version 3.0.0, https://doi.org/10.4232/1.13146. https://doi.org/10.4232/1.13146.

———. 2019a. “Eurobarometer 90.2 (2018).” GESIS Data Archive, Cologne. ZA7488 Data file Version 1.0.0, https://doi.org/10.4232/1.13289. https://doi.org/10.4232/1.13289.

———. 2019b. “Eurobarometer 91.3 (2019).” GESIS Data Archive, Cologne. ZA7572 Data file Version 1.0.0, https://doi.org/10.4232/1.13372. https://doi.org/10.4232/1.13372.

Music Streaming: Is It a Level Playing Field?

Tue, 23 Feb 2021 21:23:00 +0200

Our article, Music Streaming: Is It a Level Playing Field? is published in the February 2021 issue of CPI Antitrust Chronicle, which is fully devoted to competition policy issues in the music industry.

The dramatic growth of music streaming over recent years is potentially very positive. Streaming provides consumers with low cost, easy access to a wide range of music, while it provides music creators with low cost, easy access to a potentially wide audience. But many creators are unhappy about the major streaming platforms. They consider that they act in an unfair way, create an unlevel playing field and threaten long-term creativity in the music industry.

Our paper describes and assesses the basis for one element of these concerns, competition between recordings on streaming platforms. We argue that fair competition is restricted by the nature of the remuneration arrangements between creators and the streaming platforms, the role of playlists, and the strong negotiating power of the major labels. It concludes that urgent consideration should be given to a user-centric payment system, as well as greater transparency of the factors underpinning playlist creation and of negotiated agreements.

You can read the entire issue and the full text of our article on Competition Policy International in pdf.

Daniel Antal, co-founder of Reprex Was Selected into the 2021 Fellowship Program of the European Music Market Accelerator

Mon, 22 Feb 2021 21:23:00 +0200

Daniel Antal, co-founder of Reprex, was selected into 2021 Fellowship program of JUMP, the European Music Market Accelerator. Jump provides a framework for music professionals to develop innovative business models, encouraging the music sector to work on a transnational level. The European Music Market Accelerator composed of MaMA Festival and Convention, UnConvention, MIL, Athens Music Week, Nouvelle Prague and Linecheck support him in the development of our two, interrelated projects over the next nine months.

Our Demo Music Observatory is a demo version of the European Music Observatory based on open data, open source, automated research in open collaboration with music stakeholders. We hope that we can further develop our business model and find new users, and help the recovery of the festival and live music segment.
Listen Local is our AI system that validated third party music AI, such as Spotify’s or YouTube’s recommendation systems, and provides trustworthy, accountable, transparent alternatives for the European music industry. We hope to expand our pilot project from Slovakia to several European countries in 2021.

Reprex is committed to applying reproducible in an open collaboration with our business, scientific, policy and civil society partners, and facilitate the use of open data and open-source software.

Reprex Joins The Dutch AI Coalition

Tue, 16 Feb 2021 17:10:00 +0200

Reprex, our start-up, is based in the Netherlands and the United States that validated its early products in the Yes!Delft AI+Blockchain Lab in the Hague. In 2021, we decided to join the Dutch AI Coalition – NL AIC.

The NL AIC is a public-private partnership in which the government, the business sector, educational and research institutions, as well as civil society organisations collaborate to accelerate and connect AI developments and initiatives. The ambition is to position the Netherlands at the forefront of knowledge and application of AI for prosperity and well-being. We are continually doing so with due observance of both the Dutch and European standards and values. The NL AIC functions as the catalyst for AI applications in our country.

We are particularly looking forward to participating in the Culture working group of NLAIC, but we will also take a look at the Security, Peace and Justice and the Energy and Sustainability working groups. Reprex is committed to use and further develop AI solutions that fulfil the requirements of trustworthy AI, a human-centric, ethical, and accountable use of artificial intelligence. We are committed to develop our data platforms, or automated data observatories, and our Listen Local system in this manner. Furthermore, we are involved in various scientific collaborations that are researching ideas on future regulation of copyright and fair competition with respect to AI algorithms.

We are committed to applying reproducible in an open collaboration with our business, scientific, policy and civil society partners, and facilitate the use of open data and open-source software.

Ensuring the Visibility and Accessibility of European Creative Content on the World Market: The Need for Copyright Data Improvement in the Light of New Technologies

Sat, 13 Feb 2021 18:10:00 +0200

The majority of music sales in the world is driven by AI-algorithm powered robots that create personalized playlists, recommendations and help programming radio music streams or festival lineups. It is critically important that an artist’s work is documented, described in a way that the algorithm can work with it.

In our research paper – soon to be published – made for the Listen Local Initiative we found that 15% of Dutch, Estonian, Hungarian, or Slovak artists had no chance to be recommended, and they usually end up on Forgetify, an app that lists never-played songs of Spotify. In another project with rights management organizations, we found that about half of the rightsholders are at risk of not getting all their royalties from the platforms because of poor documentation.

But how come that distributors give streaming platforms songs that are not properly documented? What sort of information is missing for the European repertoire’s visibility? Reprex is exploring this problem in a practical cooperation with SOZA, the Slovak Performing and Mechanical Rights Society, and in an academic cooperation that involves leading researchers in the field. A manuscript co-authored Martin Senftleben, director of the Institute for Information Law in Amsterdam, and eminent researchers in copyright law and music economics, Reprex’s co-founder makes the case that Europe must invest public money to resolve this problem, because in the current scenario, the documentation costs of a song exceed the expected income from streaming platforms.

In the European Strategy for Data, the European Commission highlighted the EU’s ambition to acquire a leading role in the data economy. At the same time, the Commission conceded that the EU would have to increase its pools of quality data available for use and re-use. In the creative industries, this need for enhanced data quality and interoperability is particularly strong. Without data improvement, unprecedented opportunities for monetising the wide variety of EU creative and making this content available for new technologies, such as artificial intelligence training systems, will most probably be lost. The problem has a worldwide dimension. While the US have already taken steps to provide an integrated data space for music as of 1 January 2021, the EU is facing major obstacles not only in the field of music but also in other creative industry sectors. Weighing costs and benefits, there can be little doubt that new data improvement initiatives and sufficient investment in a better copyright data infrastructure should play a central role in EU copyright policy. A trade-off between data harmonisation and interoperability on the one hand, and transparency and accountability of content recommender systems on the other, could pave the way for successful new initiatives. Download the manuscript from SSRN

Our Slovak Demo Music Database project is a best example for this. We started systematically collect publicly available information from Slovak artists (in our write-in process) and ask them to give GDPR-protected further data (in our opt-in process) to create a comprehensive database that can help recommendation engines as well as market-targeting or educational AI apps.

We believe that one of the problems of current AI algorithms that they solely or almost only work with English language documentation, putting other, particularly small language repertoires at risk of being buried below well-documented music mainly arriving from the United States.

We are looking for rightsholders and their organizations, artists, researchers to work with us to find out how we can increase the visibility of European music.

Reprex introduction in IVIR

Tue, 02 Feb 2021 10:10:00 +0100

IViRtual 9 April 2021

Data Curation

Thu, 21 Jan 2021 00:00:00 +0000

If you cannot find the right data for your policy evaluation, your consulting project, your PhD thesis, your market research, or your scientific research project, it does not mean that the data does not exist, or that it is not available for free. In our experience, up to 95% of available open data is never used, because potential users do not realize it exists or do not know how to access it.

Every day, thousands of new datasets become available via the EU open data regime, freedom of information legislation in the United States and other jurisdictions, or open science and scientific reproducibility requirements — but as these datasets have been packaged or processed for different primary, original uses, they often require open data experts to locate them and adapt them to a usable form for reuse in business, scientific, or policy research.

The creative and cultural industries often do not participate in government statistics programs because these industries are typically comprised of microenterprises that are exempted from statistical reporting and that file only simplified financial statements and tax returns. This means that finding the appropriate private or public data sources for creative and cultural industry uses requires particularly good data maps.

Data curation means that we are continuously mapping potential data sources and sending requests to download and quality test the most current data sources. Our CEEMID project has produced several thousand indicators, of which a few dozen are available in our Demo Music Observatory.If you have specific data needs for a scientific research, policy evaluation, or business project, we can find and provide the most suitable, most current, and best value data for analysis or for ethical AI applications.

Data Processing

Thu, 21 Jan 2021 00:00:00 +0000

Data analysts spend 80% of their time on data processing, even though computers can perform these task much faster, with far less errors, and they can document the process automatically. Data processing can be shared: an analyst in a company and an analyst in an NGO does not have to reprocess the very same data twice*

See our blogpost How We Add Value to Public Data With Imputation and Forecasting?.

See our blogpost about the Data Sisyphus blogpost.

We have a better solution. You can always rely on our API to import directly the latest, best data, but if you want to be sure, you can use our regular backups on Zenodo. Zenodo is an open science repository managed by CERN and supported by the European Union. On Zenodo, you can find an authoritative copy of our indicator (and its previous versions) with a digital object identifier, for example, 10.5281/zenodo.5652118. These datasets will be preserved for decades, and nobody can manipulate them. You cannot accidentally overwrite them, and we have no backdoor access to modify them.

Data-as-Service

Thu, 21 Jan 2021 00:00:00 +0000

We want to ensure that individual researchers, artists, and professionals, as well as NGOs and small and large organizations can benefit equally from big data in the age of artificial intelligence.

Big data creates inequality and injustice because it is only the big corporations, big government agencies, and the biggest, best endowed universities that can finance long-lasting, comprehensive data collection programs. Big data, and large, well-processed, tidy, and accurately imputed datasets allow them to unleash the power of machine learning and AI. These large entities are able to create algorithms that decide the commercial success of your product and your artwork, giving them a competitive edge against smaller competitors while helping them evade regulations.

Check out our iotables software that helps the use of national accounts data from all EU members states to create economic direct, indirect and induced economic impact calculation, such as employment multipliers or GVA affects of various cultural and creative economy policies.

Check out our regions software that helps the harmonization of various European and African standardized surveys.

Check out our retroharmonize software that helps the harmonization of various European and African standardized surveys.

Economy Data Observatory

Thu, 21 Jan 2021 00:00:00 +0000

Big data and automation create new inequalities and injustices and has a potential to create a jobless growth. Our Economy Observatory is a fully automated, open source, open data observatory that produces new indicators from open data sources and experimental big data sources, with authoritative copies and a modern API.

Our observatory is monitoring the European economy to protect the consumers and the small companies from unfair competition both from data and knowledge monopolization and robotization. We take a critical SME-, intellectual property policy and competition policy point of view automation, robotization, and the AI revolution on the service-oriented European social market economy.

We would like to create early-warning, risk, economic effect, and impact indicators that can be used in scientific, business and policy contexts for professionals who are working on re-setting the European economy after a devastating pandemic and in the age of AI. We would like to map data between economic activities (NACE), antitrust markets, and sub-national, regional, metropolitian area data.

Get involved in services: our ongoing projects, team of contributors, open-source libraries and use our data for publications. See some use cases.

Follow news about us or the more comprehensive Data & Lyrics blog.

Download our competition presentation

Our Product/Market Fit was validated in the world’s 2nd ranked university-backed incubator program, the Yes!Delft AI Validation Lab.

Demo Slovak Music Database

Thu, 17 Dec 2020 17:10:00 +0200

We are finalizing our first local recommendation system, Listen Local Slovakia, and the accompanying Demo Slovak Music Database. Our aim is

Show how the Slovak repertoire is seen by media and streaming platforms
What are the possibilities to give greater visibility to the Slovak repertoire in radio and streaming platforms
What are the specific problems why certain artists and music is almost invisible.

In the next year, we would like to create a modern, comprehensive national music database that serves music promotion in radio, streaming, live music within Slovakia and abroad.

To train our locally relevant, alternative recommendation system, we filled the Demo Slovak Music Database from two sources. In the opt-in process we asked artists to participate in Listen Local, and we selected those artists who opted in from Slovakia, or whose language is Slovak. In the write-in process we collected publicly available data from other artists that our musicology team considered to be Slovak, mainly on the basis of their language use, residence, and other public biographical information. The following artists form the basis of our experiment. (If you want to be excluded from the write-in list, write to us, or you want to be included, please, fill out this form.)

Click here to view the table on a separate page

Modern recommendation systems usually rely on data provided by artists or their representatives, data on who and how is listening to their music, and what music is listened to by the audience of the artists, and certain musicological features of the music. Usually they collect data from various data sources, but these data sources are mainly English language sources.

The problem with these recommendation systems is that they do not help music discovery, and make starting new acts very difficult. Recommendation systems tend to help already established artists, and artists whose work is well described in the English language.

Our alternative recommendation system is a utility-based system that gives a user-defined priority to artists released in Slovakia, or artists identified as Slovak, or both. The system can be extended for lyrics language priorities, too. Currently, our app is demonstration to provide a more comprehensive database-driven tool that can support various music discovery, recommendation or music export tools. Our Feasibility Study to build such tools and our Demo App is currently under consultation with Slovak stakeholders.

Listen Local is developing transparent algorithms and open source solutions to find new audiences for independent music. We want to correct the injustice and inherent bias of market leading big data algorithms. If you want your music and audience to be analysed in Listen Local, fill this form in. We will include you in our demo application for local music recommendations and our analysis to be revealed in December.

Product/Market Fit Validation in Yes!Delft

Fri, 25 Sep 2020 15:31:39 +0200

We would like to validate our product market/fit in two segments, business/policy research and scientific research, with a supporting role given to data journalism. Because we want to follow a bootstrapping strategy, we must focus on those clients where we find the highest value proposition, which is of course easier said than done. We see much interest in our offering from other continents, therefore we truly welcome the opportunity that we can do this on a truly global business canvas in one of the worlds’ top five incubators, the number 2 university-backed incubator in the world, second to none in Europe, in the Yes!Delft AI+Blockchain Validation Lab.

In Europe hundreds of thousands of microenterprises, such as record labels, video producers or book publishers are facing data and AI giants like Google’s YouTube, Apple Music, Spotify, Netflix or Amazon. If the recommendation engines of these giants do not recommend their songs, films or books, then their investments are doomed to fail, because about half of the global sales are driven by AI algorithms. When they make a claim for the missing money, they will immediately find themselves in a dispute with gigabytes of data that they can only handle with a data scientist, even though they do not even have an IT professional or an HR professional to make the hire.

An awful lot of money, creativity and real values are at stake, and we want to be on the creator’s side, their technician’s side, their manager’s side when they want to get a fair share from the pie and they want to help these industry leader to make the pie grow.

The UNESCO and the EU have been promoting as an organizational solution the fragmentation problem with the so-called data observatories that are pooling the business, policy, and scientific research needs of various domains, like music. This is an idea that we really like, and we believe that our research automation solutions can help these observatories to grow faster as ecosystems, create better quality and more timely data and research products and a far lower cost.

We define ourselves as a reproducible research company inspired by the philosophy of open collaboration, based on open-source software and open data. We want to explore various revenue models around these ideas.

We are not committed to open source licensing if more permissive licensing policies provide us with better opportunities.
We would like to explore various data-as-service models, because we do not want to be locked into the position of cheap open data vendors.
We want to deploy AI applications that really help earning money in these sectors with playlisting, recommendation engines, forecasting applications, or royalty valuations, because our open collaboration approach brings up enough data sooner to than its alternatives, because it manages inherent conflicts of interests, fragmentation, and decentralization better than hierarchical solutions.

Timeline

In January CEEMID reached its peak: we introduced a 12-country reproducible research project made with only freelancers in Brussels, presented as best use case of evidence-based policy design.
In February Daniel visited the Yes!Delft Co-Lab to find out who would be the best co-founder to re-launch CEEMID as an enterprise.
In April we started to release our data as open data for validation.
One month ago we started-up.
Then we launched the music.dataobservatory.eu project.
A few other data observatories.

Bonus:

Palato in the Hague, where we took our selfie and had an absolutely amazing dinner after the pitch. Check them out!

Digital Music Observatory

Tue, 15 Sep 2020 00:00:00 +0000

The Digital Music Observatory is a fully automated, open source, open data observatory that links public datasets in order to provide a comprehensive view of the European music industry. The DMO produces key business and policy indicators that enable the growth of music business strategies and national music policies in a way that works both for music lover audiences and the creative enterprises of the sector, and contributes to a more competitive, fair and sustainable European music ecosystem.

Download our introduction.

Visit the Digital Music Observatory

Try the Digital Music Observatory API

Connect on LinkedIn

Our data pillars are following the structure laid out in the Feasibility study for the establishment of a European Music Observatory: Music Economy; Diversity & Circulation; Music & Society and Innovation - innovative data applications

Follow news about us or the more comprehensive Data & Lyrics blog.

Music is one of the most data-driven service industries where the majority of sales are already made by AI-driven autonomous systems. The DMO is a fully-functional service that can function as a testing ground of the European Data Strategy, showcasing the ways in which the music industry is affected by the problems that the Digital Services Act and the Trustworthy AI initiatives attempt to regulate. If these policies will work for the European microenterprise-dominated, complex and fragile European music ecosystem, then they are likely to make Europe fit for the digital age.

Creating An Automated Data Observatory

Fri, 11 Sep 2020 16:00:39 +0200

We are building data ecosystems, so called observatories, where scientific, business, policy and civic users can find factual information, data, evidence for their domain. Our open source, open data, open collaboration approach allows to connect various open and proprietary data sources, and our reproducible research workflows allow us to automate data collection, processing, publication, documentation and presentation.

Our scripts are checking data sources, such as Eurostat’s Eurobase, Spotify’s API and other music industry sources every day for new information, and process any data corrections or new disclosure, interpolate, backcast or forecast missing values, make currency translations and unit conversions. This is shown illustrated with an earlier post.

For direct access to the file visit this link.

In the video we show automated the creation of an observatory website with well-formatted, statistical data dissemination, a technical document in PDF and an ebook can be automated. In our view, our technology is particularly useful technology in business and scientific researech projects, where it is important that always the most timely and correct data is being analyzed, and remains automatically documented and cited. We are ready deploy public, collaborative, or private data observatories in short time.

Data processing costs can be as high as 80% for any in-house AI deployment project. We work mainly with organization that do not have in house data science team, and acquire their data anyway from outside the organization. In their case, this rate can be as high as 95%, meaning that getting and processing the data for deploying AI can be 20x more expensive than the AI solution itself.

AI solutions require a large amount of standardized, well processed data to learn from. We want to radically decrease the cost of data acquisition and processing for our users so that exploiting AI becomes in their reach. This is particularly important in one of our target industries, the music industries, where most of the global sales is algorithmic and AI-driven. Artists, bands, small labels, publishers, even small country national associations cannot remain competitive if they cannot participate in this technological revolution.

We started our operations on 1 September 2020 on the basis of CEEMID, a pan-European data observatory that created about 2000 music and creative industry indicators for its users. In the coming days, we are gradually opening up about 50 music industry and 50 broader creative industry indicators in a fully reproducible workflow, with daily re-freshed, re-processed, well-formatted and documented indicators for business and policy decisions.

We would like to validate this approach in one of the world’s most prestigious university-backed incubator programs, in the Yes!Delft AI/Blockchain Validation Lab.

Video credits

Data acquisition and processing: Daniel Antal, CFA and Marta Kołczyńska, PhD (survey data).
Documentation automation: Sandor Budai
Video art: Line Matson
Music: Moon Moon Moon.

retroharmonize R package for survey harmonization

Tue, 25 Aug 2020 00:00:00 +0000

Retrospective data harmonization

The aim of retroharmonize is to provide tools for reproducible retrospective (ex-post) harmonization of datasets that contain variables measuring the same concepts but coded in different ways. Ex-post data harmonization enables better use of existing data and creates new research opportunities. For example, harmonizing data from different countries enables cross-national comparisons, while merging data from different time points makes it possible to track changes over time.

Retrospective data harmonization is associated with challenges including conceptual issues with establishing equivalence and comparability, practical complications of having to standardize the naming and coding of variables, technical difficulties with merging data stored in different formats, and the need to document a large number of data transformations. The retroharmonize package assists with the latter three components, freeing up the capacity of researchers to focus on the first.

Specifically, the retroharmonize package proposes a reproducible workflow, including a new class for storing data together with the harmonized and original metadata, as well as functions for importing data from different formats, harmonizing data and metadata, documenting the harmonization process, and converting between data types. See here for an overview of the functionalities.

The new labelled_spss_survey() class is an extension of haven’s labelled_spss class. It not only preserves variable and value labels and the user-defined missing range, but also gives an identifier, for example, the filename or the wave number, to the vector. Additionally, it enables the preservation – as metadata attributes – of the original variable names, labels, and value codes and labels, from the source data, in addition to the harmonized variable names, labels, and value codes and labels. This way, the harmonized data also contain the pre-harmonization record. The stored original metadata can be used for validation and documentation purposes.

The vignette Working With The labelled_spss_survey Class provides more information about the labelled_spss_survey() class.

In Harmonize Value Labels we discuss the characteristics of the labelled_spss_survey() class and demonstrates the problems that using this class solves.

We also provide three extensive case studies illustrating how the retroharmonize package can be used for ex-post harmonization of data from cross-national surveys:

The creators of retroharmonize are not affiliated with either Afrobarometer, Arab Barometer, Eurobarometer, or the organizations that designs, produces or archives their surveys.

We started building an experimental APIs data is running retroharmonize regularly and improving known statistical data sources. See: Digital Music Observatory, Green Deal Data Observatory, Economy Data Observatory.

Citing the data sources

Our package has been tested on three harmonized survey’s microdata. Because retroharmonize is not affiliated with any of these data sources, to replicate our tutorials or work with the data, you have download the data files from these sources, and you have to cite those sources in your work.

Afrobarometer data: Cite Afrobarometer Arab Barometer data: cite Arab Barometer. Eurobarometer data: The Eurobarometer data Eurobarometer raw data and related documentation (questionnaires, codebooks, etc.) are made available by GESIS, ICPSR and through the Social Science Data Archive networks. You should cite your source, in our examples, we rely on the GESIS data files.

Citing the retroharmonize R package

For main developer and contributors, see the package homepage.

This work can be freely used, modified and distributed under the GPL-3 license:

citation("retroharmonize")
#> 
#> To cite package 'retroharmonize' in publications use:
#> 
#>   Daniel Antal (2021). retroharmonize: Ex Post Survey Data
#>   Harmonization. R package version 0.1.17.
#>   https://retroharmonize.dataobservatory.eu/
#> 
#> A BibTeX entry for LaTeX users is
#> 
#>   @Manual{,
#>     title = {retroharmonize: Ex Post Survey Data Harmonization},
#>     author = {Daniel Antal},
#>     year = {2021},
#>     doi = {10.5281/zenodo.5006056},
#>     note = {R package version 0.1.17},
#>     url = {https://retroharmonize.dataobservatory.eu/},
#>   }

Contact

For contact information, contributors, see the package homepage.

Code of Conduct

Please note that the retroharmonize project is released with a Contributor Code of Conduct. By contributing to this project, you agree to abide by its terms.

Click the Cite button above to demo the feature to enable visitors to import publication metadata into their reference management software.

regions R package to create sub-national statistical indicators

Wed, 03 Jun 2020 17:00:00 +0000

Installation

You can install the development version from GitHub with:

devtools::install_github("rOpenGov/regions")

or the released version from CRAN:

install.packages("devtools")

regions currently takes care of 20,000 sub-divisional boundary changes in Europe since 1999. Comparing departments of France in 2013, with 2007 vojvodinas of Poland and 2018 megyék in Hungary? This extremely errorprone work is automated, as a result, you can compare 110-260 regions for far better analysis. regions was downloaded about 600 researchers in the first month after release.

You can review the complete package documentation on regions.dataobservatory.eu. If you find any problems with the code, please raise an issue on Github. Pull requests are welcome if you agree with the Contributor Code of Conduct

If you use regions in your work, please cite the package.

Motivation

Working with sub-national statistics has many benefits. In policymaking or in social sciences, it is a common practice to compare national statistics, which can be hugely misleading. The United States of America, the Federal Republic of Germany, Slovakia and Luxembourg are all countries, but they differ vastly in size and social homogeneity. Comparing Slovakia and Luxembourg to the federal states or even regions within Germany, or the states of Germany and the United States can provide more adequate insights. Statistically, the similarity of the aggregation level and high number of observations can allow more precise control of model parameters and errors.

The advantages of switching from a national level of the analysis to a sub-national level comes with a huge price in data processing, validation and imputation. The package Regions aims to help this process.

Sub-national Statistics Have Many Challenges

Frequent boundary changes: as opposed to national boundaries, the territorial units, typologies are often change, and this makes the validation and recoding of observation necessary across time. For example, in the European Union, sub-national typologies change about every three years and you have to make sure that you compare the right French region in time, or, if you can make the time-wise comparison at all.
Hierarchical aggregation and special imputation: missingness is very frequent in sub-national statistics, because they are created with a serious time-lag compared to national ones, and because they are often not back-casted after boundary changes. You cannot use standard imputation algorithms because the observations are not similarly aggregated or averaged. Often, the information is seemingly missing, and it is present with an obsolete typology code.

Package functionality

Generic vocabulary translation and joining functions for geographically coded data
Keeping track of the boundary changes within the European Union between 1999-2021
Vocabulary translation and joining functions for standardized European Union statistics
Vocabulary translation for the ISO-3166-2 based Google data and the European Union
Imputation functions from higher aggregation hierarchy levels to lower ones, for example from NUTS1 to NUTS2 or from ISO-3166-1 to ISO-3166-2 (impute down)
Imputation functions from lower hierarchy levels to higher ones (impute up)
Aggregation function from lower hierarchy levels to higher ones, for example from NUTS3 to NUTS1 or from ISO-3166-2 to ISO-3166-1 (aggregate; under development)
Disaggregation functions from higher hierarchy levels to lower ones, again, for example from NUTS1 to NUTS2 or from ISO-3166-1 to ISO-3166-2 (disaggregate; under development)

Vignettes / Articles

Feedback?

Raise and issue on Github or get in touch. Downloaders from CRAN:

Click the Cite button above to demo the feature to enable visitors to import publication metadata into their reference management software.

iotables R package for working with symmetric input-output tables

Wed, 03 Jun 2020 00:00:00 +0000

iotables processes all the symmetric input-output tables of the EU member states, and calculates direct, indirect and induced effects, multipliers for GVA, employment, taxation. These are important inputs into policy evaluation, business forecasting, or granting/development indicator design. iotables is used by about 800 experts around the world.

Code of Conduct

Please note that the iotables project is released with a Contributor Code of Conduct. By contributing to this project, you agree to abide by its terms.

Click the Cite button above to demo the feature to enable visitors to import publication metadata into their reference management software.

Slides

Tue, 21 Jan 2020 00:00:00 +0000

Data Curation With Reprex

Reprex | Data Curation

Presentation Controls

Next: Right Arrow or Space
Previous: Left Arrow
Start: Home
Finish: End
Overview: Esc
Speaker notes: S
Fullscreen: F
Zoom: Alt + Click
PDF Export: E

Data Curation

If you do not find the data for your research, reporting or evaluation project, it does not mean that the data does not exist. It does not mean that the data is not available for free.

Fragments

Make content appear incrementally

One Two Three

A fragment can accept two optional parameters:

class: use a custom style (requires definition in custom CSS)
weight: sets the order in which a fragment appears

Speaker Notes

Add speaker notes to your presentation

{{% speaker_note %}}
- Only the speaker can read these notes
- Press `S` key to view
{{% /speaker_note %}}

Press the S key to view the speaker notes!

Themes

black: Black background, white text, blue links (default)
white: White background, black text, blue links
league: Gray background, white text, blue links
beige: Beige background, dark text, brown links
sky: Blue background, thin dark text, blue links

night: Black background, thick white text, orange links
serif: Cappuccino background, gray text, brown links
simple: White background, black text, blue links
solarized: Cream-colored background, dark green text, blue links

Custom Slide

Customize the slide style and background

{{< slide background-image="/media/boards.jpg" >}}
{{< slide background-color="#0000FF" >}}
{{< slide class="my-style" >}}

Custom CSS Example

Let’s make headers navy colored.

Create assets/css/reveal_custom.css with:

.reveal section h1,
.reveal section h2,
.reveal section h3 {
  color: navy;
}

Questions?

Ask

Documentation

Slides

Tue, 21 Jan 2020 00:00:00 +0000

Data Curation With Reprex

Reprex | Data Curation

Presentation Controls

Next: Right Arrow or Space
Previous: Left Arrow
Start: Home
Finish: End
Overview: Esc
Speaker notes: S
Fullscreen: F
Zoom: Alt + Click
PDF Export: E

Novel Data Products

See our 100,000 Opinions on the Most Pressing Global Problem blogpost.

Official statistics at the national and European levels follow legal regulations, and in the EU, compromises between member states. New policy indicators often appear 5-10 years after demand appears. We employ the same methodology, software, and often even the same data that Eurostat might use to develop policy indicators, but we do not have to wait for a political and legal consensus to create new datasets.

Data Curation

If you do not find the data for your research, reporting or evaluation project, it does not mean that the data does not exist. It does not mean that the data is not available for free.

Fragments

Make content appear incrementally

One Two Three

A fragment can accept two optional parameters:

class: use a custom style (requires definition in custom CSS)
weight: sets the order in which a fragment appears

Themes

black: Black background, white text, blue links (default)
white: White background, black text, blue links
league: Gray background, white text, blue links
beige: Beige background, dark text, brown links
sky: Blue background, thin dark text, blue links

night: Black background, thick white text, orange links
serif: Cappuccino background, gray text, brown links
simple: White background, black text, blue links
solarized: Cream-colored background, dark green text, blue links

Custom Slide

Customize the slide style and background

{{< slide background-image="/media/boards.jpg" >}}
{{< slide background-color="#0000FF" >}}
{{< slide class="my-style" >}}

Custom CSS Example

Let’s make headers navy colored.

Create assets/css/reveal_custom.css with:

.reveal section h1,
.reveal section h2,
.reveal section h3 {
  color: navy;
}

Questions?

Ask

Documentation

Slides

Tue, 21 Jan 2020 00:00:00 +0000

Data Curation With Reprex

Reprex | Data Curation

Presentation Controls

Next: Right Arrow or Space
Previous: Left Arrow
Start: Home
Finish: End
Overview: Esc
Speaker notes: S
Fullscreen: F
Zoom: Alt + Click
PDF Export: E

Data Curation

If you do not find the data for your research, reporting or evaluation project, it does not mean that the data does not exist. It does not mean that the data is not available for free.

Fragments

Make content appear incrementally

One Two Three

A fragment can accept two optional parameters:

class: use a custom style (requires definition in custom CSS)
weight: sets the order in which a fragment appears

Speaker Notes

Add speaker notes to your presentation

{{% speaker_note %}}
- Only the speaker can read these notes
- Press `S` key to view
{{% /speaker_note %}}

Press the S key to view the speaker notes!

Themes

black: Black background, white text, blue links (default)
white: White background, black text, blue links
league: Gray background, white text, blue links
beige: Beige background, dark text, brown links
sky: Blue background, thin dark text, blue links

night: Black background, thick white text, orange links
serif: Cappuccino background, gray text, brown links
simple: White background, black text, blue links
solarized: Cream-colored background, dark green text, blue links

Custom Slide

Customize the slide style and background

{{< slide background-image="/media/boards.jpg" >}}
{{< slide background-color="#0000FF" >}}
{{< slide class="my-style" >}}

Custom CSS Example

Let’s make headers navy colored.

Create assets/css/reveal_custom.css with:

.reveal section h1,
.reveal section h2,
.reveal section h3 {
  color: navy;
}

Questions?

Ask

Documentation

Slides

Tue, 21 Jan 2020 00:00:00 +0000

Research Automation With Reprex

Reprex | Research Automation

Presentation Controls

Next: Right Arrow or Space
Previous: Left Arrow
Start: Home
Finish: End
Overview: Esc
Speaker notes: S
Fullscreen: F
Zoom: Alt + Click
PDF Export: E

Research Automation

We believe that if you have already done the same data processing or controlling step twice in your team, the problem will show up the third and the fourth time, too.

If your junior researchers have downloaded the same dataset at least twice, or made the same processing steps in Excel or Power BI already twice, then it is high-time to automate the process, because each manual process is time-consuming and error-prone.
For your senior researchers, automation of data validation and unit-testing can save equally many hours.

Fragments

Make content appear incrementally

One Two Three

A fragment can accept two optional parameters:

class: use a custom style (requires definition in custom CSS)
weight: sets the order in which a fragment appears

Speaker Notes

Add speaker notes to your presentation

{{% speaker_note %}}
- Only the speaker can read these notes
- Press `S` key to view
{{% /speaker_note %}}

Press the S key to view the speaker notes!

Themes

black: Black background, white text, blue links (default)
white: White background, black text, blue links
league: Gray background, white text, blue links
beige: Beige background, dark text, brown links
sky: Blue background, thin dark text, blue links

night: Black background, thick white text, orange links
serif: Cappuccino background, gray text, brown links
simple: White background, black text, blue links
solarized: Cream-colored background, dark green text, blue links

Custom Slide

Customize the slide style and background

{{< slide background-image="/media/boards.jpg" >}}
{{< slide background-color="#0000FF" >}}
{{< slide class="my-style" >}}

Custom CSS Example

Let’s make headers navy colored.

Create assets/css/reveal_custom.css with:

.reveal section h1,
.reveal section h2,
.reveal section h3 {
  color: navy;
}

Questions?

Ask

Documentation

Slides

Tue, 21 Jan 2020 00:00:00 +0000

Trustworthy AI With Reprex

Reprex | Data Curation

Presentation Controls

Next: Right Arrow or Space
Previous: Left Arrow
Start: Home
Finish: End
Overview: Esc
Speaker notes: S
Fullscreen: F
Zoom: Alt + Click
PDF Export: E

Trustworthy AI

The European Union has adopted the Ethics Guidelines for Trustworthy AI with seven important principles:

Human agency and oversight
Technical robustness and safety
Privacy and Data governance
Transparency
Diversity, non-discrimination and fairness
Societal and environmental well-being
Accountability

In the EU, in the US or in the UK creative industries usually do not have access to trustworthy AI in 2021.

Our Focus

Make content appear incrementally

Transparency: we use open data, open source software, and we make sure that our clients understand what is happening with their products
Diversity: we know that a Slovak, Hungarian or Estonian song has less chances than an American one. We see that many corporate algorithms reinforce racism and patriarchy.
Societal well-being: we understand that the algorithm is maximizing the well-being and profit of the entity that trains it.
Accountability: algorithm-driven robotic sales must be held accountable, and must comply with consumer-protection, competition-protecion and non-discrimination rules.

A fragment can accept two optional parameters:

class: use a custom style (requires definition in custom CSS)
weight: sets the order in which a fragment appears

Speaker Notes

Add speaker notes to your presentation

{{% speaker_note %}}
- Only the speaker can read these notes
- Press `S` key to view
{{% /speaker_note %}}

Press the S key to view the speaker notes!

Themes

black: Black background, white text, blue links (default)
white: White background, black text, blue links
league: Gray background, white text, blue links
beige: Beige background, dark text, brown links
sky: Blue background, thin dark text, blue links

night: Black background, thick white text, orange links
serif: Cappuccino background, gray text, brown links
simple: White background, black text, blue links
solarized: Cream-colored background, dark green text, blue links

Custom Slide

Customize the slide style and background

{{< slide background-image="/media/boards.jpg" >}}
{{< slide background-color="#0000FF" >}}
{{< slide class="my-style" >}}

Custom CSS Example

Let’s make headers navy colored.

Create assets/css/reveal_custom.css with:

.reveal section h1,
.reveal section h2,
.reveal section h3 {
  color: navy;
}

Questions?

Ask

Documentation

Market size of the re-usable public sector information in Hungary

Tue, 15 Dec 2009 00:00:00 +0000

Original title in Hungarian: A közintézmények újrahasznosítható információinak piaca Magyarországon.

Mon, 01 Jan 0001 00:00:00 +0000

Robin Nagy

Mon, 01 Jan 0001 00:00:00 +0000

Robin is a freelancer working on projects using emerging technologies. Main areas of experties are Business Development, Project Management, Innovation, Governance and Corporate culture. Recent interest: Exponential technologies, Exponential business.

Automated Data Observatories

How We Add Value to Public Data With Imputation and Forecasting

Why is data missing?

What can we improve?

Can you trust our data?

Avoid the data Sisyphus

How We Add Value to Public Data With Better Curation And Documentation?

Findable Data

Accessible Data

Interoperability

Reuse

Digital Music Observatory on MaMA 2021

Use Cases

Use Cases 2

Questions?

Open Data - The New Gold Without the Rush

Use Cases

Use Cases 2

Questions?

Digital Music Observatory on the MaMA Convention 2021

Presentation Slides

Crunchconf: Open Data, New Gold Without the Rush

Summary

Presentation Slides

CCSI Data Observatory

Creative & Cultural Sectors Industries Data Observatory

The Data Sisyphus

The hidden cost item behind the unbillable hours

Our observatory automatically processes and documents the data

The value of metadata

Green Deal Data Observatory

Better, Bigger, Faster, More

Increase Your Impact, Avoid Old Mistakes

Ethical Big Data for All

Open Collaboration

Metadata

The hidden cost item

Map your data: bibliographis, catalogues, codebooks, versioning

What happened with the data before?

Survey Harmonization

Including Indicators from Arab Barometer in Our Observatory

Further plans

Use our software

Tutorial to work with the Arab Barometer survey data

Experimental API

Join our open source effort

Competition Data Observatory

Better, Bigger, Faster, More

Increase Your Impact, Avoid Old Mistakes

Ethical Big Data for All

Open Collaboration

Open Data - The New Gold Without the Rush

Music Creators’ Earnings in the Streaming Era

Justified and Unjustified Differences in Earnings

Justified Difference & Changes Over Time

Solidarity & Equitable Remuneration

The Size of the Pie and the Distribution of the Pie

Join us

Footnote References

Analyze Locally, Act Globally: New regions R Package Release

Get the Package

Join us

Open Data is Like Gold in the Mud Below the Chilly Waves of Mountain Rivers

Join us

Trustworthy AI: Check Where the Machine Learning Algorithm is Learning From

Read More on Data & Lyrics

Join Copernicus Climate Data Store Data with Socio-Economic and Opinion Poll Data

Register to the Copernicus Climate Data Store

Get the Data

Effective Leaf Area Index

Get The Map

Data Integrity

Join us

Economic and Environment Impact Analysis, Automated for Data-as-Service

Accessing and tidying the data programmatically

Let’s seen an example

Vignettes

Environmental Impact Analysis

rOpenGov and the EU Datathon Challenges

Data API