Lesson 1
Data Ideas, Formats,
and Sources
and Sources
Project Ideas
Data Types and Formats
Excel Interface and Formulas
Data Types and Formats
Excel Interface and Formulas
Algorithm
- Idea + Data
- Data Cleaning & Processing
- Analysis
- External validation — searching for proofs outside of data (experts, media etc.) that your conclusions are sane and plausible
- Visualisation
- Text
- Front-end and publishing
Data is a representation of information in a formalised form suitable for transmission, interpretation and processing by humans or computers.
Ιnformation hierarchy:
Signal → Data → Information → Knowledge → Wisdom
Data is formalised signal.
Information is organized or structured data, which has been processed in such a way that the information now has relevance for a specific purpose or context, and is therefore meaningful, valuable, useful and relevant.
In other words, information is a human-understandable message obtained by processing and analysing data; data with meaningful context.
Knowledge is meaningful information that allows us to form an understanding of a subject. We have knowledge when we know what to do with the information.
Understanding is the ability to see the logical connections within the acquired knowledge and apply it to understand the world around us. When we consider the possibilities and limits of applicability of knowledge, we move to the level of understanding.
Wisdom is knowledge and understanding put into action: the insights we gain help us decide what is the best future course of action.
Information made super-useful. Wisdom is knowing the right things to do.
Idea + Data Strategies
- From Idea to Data.
Then we look for a suitable data source, such as the worldwide gross domestic product statistics on the UNData website.
2. From Data To Idea.
First we find an interesting dataset - for example, what is the average gross domestic product per capita in each country.
And then, scrutinising the data, we find insights and an idea for an article, for example:
‘Where do the richest people live: the United States, Monaco or Singapore?’
Ответ: вагадане — золотодобыча, рыбная ловля, энергетика.
3. Soup Strategy: Mixture of Different Topics
- Topic 1: Folklore
- Topic 2: Road Traffic Accidents
- Article: "Evil сrossroads: is it a random coincidence that, according to folklore, witches' initiation ritual takes place, where the road accidents mortality rate is the highest?"
🌻 HIGHLY IMPORTANT!
You shouldn't start working on an article until you are sure that the data you need exists and there is enough data for your research.
It doesn't matter whether the data is big or small: there must be enough.
Formats
- xlsx — well-known format for Microsoft Excel documents
- csv/ tsv — Comma-Separated Values/ Tab-Separated Values
- web table — from Wikipedia etc. (i.e. List of Nobel laureates)
- pdf table — tables in pdf-files
- xml — Εxtensible Μarkup Language — tree-structured data
- json — JavaScript Object Notation — a tree with name-value pairs
- geojson — the same but with geographic coordinates
- unstructured data — anything that needs to be preprocessed for computational analysis
🌻 HIGHLY IMPORTANT!
Ready infographics (pictures with some numbers or graphs) are NOT data!
Big & Open ❤️
Big data — 7 Vs:
- Volume — big size.
- Variety — various types, formats, structure, sources and nature of data.
- Velocity — data is automatically generated and processed at high speed.
- Veracity — reliability of data quality and value
- Value — worth analysing.
- Variability — the meaning of particular data instances is changeable, the changes in the outer context and methodologies affect the data and its interpretation.
- Visualisation — key insights can be summed up in a picture.
Open data — freely available machine-readable data to everyone to use.
Synthetic data is computer-generated data. It resembles real data, is created using models and algorithms, but is entirely fictitious. It does not contain personal information. It is suitable for training neural networks, but not suitable for data journalism investigations.
Your data — self-tracking — see Quantified Self (+ r/QuantifiedSelf)
- eg. income and expenses,
- steps a day etc.
Sample projects:
- "Who am I?" by Anna Snellman
- Made to Measure
Can you imagine a variable that may be both qualitative and quantitative in different situations?
Yes
Yes
Tutorials
All videos have subtitles, just turn them on in the player.
If you have some difficulties with English, check out the Russian version of these videos.
If you have some difficulties with English, check out the Russian version of these videos.
USEFUL DATASETS
Folder with all needed datasets
Folder with all needed datasets
- TSV — Sample tsv
- CSV — Happy together
- CSV — New Year’s Resolutions in 2021 and 2022 (from the folder)
- JSON — Countries
- XML — Museums and Galleries
How to open a csv-file
How to open a json-file
You can't directly import a json or xml file to Google Spreadsheets.
1. Convert your file to CSV, eg. using this website —
JSON to CSV, XML to CSV.
2. Import the result to Google Spreadsheets as a csv-file.
Do you have an old version of Microsoft Office (2013 and earlier)?
Use the same strategy — convert your file to csv and then import to Excel.
Sometimes a tree-structured files like json or xml give you a whole bunch of errors instead of a usable table.
Maybe it's not your fault, maybe the file itself is ill-formed.
Check the validity of your json here and of your xml here.
1. Convert your file to CSV, eg. using this website —
JSON to CSV, XML to CSV.
2. Import the result to Google Spreadsheets as a csv-file.
Do you have an old version of Microsoft Office (2013 and earlier)?
Use the same strategy — convert your file to csv and then import to Excel.
Sometimes a tree-structured files like json or xml give you a whole bunch of errors instead of a usable table.
Maybe it's not your fault, maybe the file itself is ill-formed.
Check the validity of your json here and of your xml here.
How to open an xml
Now there are:
If you need me to include another language, just ask 💚
- English
- Russian
- Simplified Chinese
- Spanish
- Turkish
- Azerbaijani
- Latvian
If you need me to include another language, just ask 💚
Data Sources
You can usually find the data you need by simply googling ‘data theme’ or ‘statistics’ such as environmental data. However, here are a few key sites where you can find lots of useful datasets.
Open government data — global level
Non-governmental aggregators — global level
Open government data — global level
- UNData — United Nations
- World Health Organization
- World Bank
- UNCTADStat — United Nation Conference of Trade and Development
- UNESCO: Overall Statistics and World Inequality Database on Education
- Eurostat Database — European Union
- Consulting reports, e.g. McKinsey & Company
Non-governmental aggregators — global level
- Dateno — worldwide data search service created by Russian developers.
- Our World in Data.
- Gapminder.
- Statista.com — 🌻 Important! Some statistics require a paid subscription — they have a green-white cross at the top right corner of an icon. Statistics without this cross are free.
- OCCRP Aleph — for investigative reporting (with VPN in Russia).
- Data.world.
- Kaggle — here you might stumble across something silly, like "Best Books Ever", "Reddit Dad Jokes", "Naughty or Nice List 2021", "French Second Hand Cars", "List of Colours" — look through Trending Datasets.
- Search trends analytics: Google Trends and Yandex Wordstat.
Российские данные
- Rosstat — you may download the data and request information.
- DataCatalogs — aggregator of Russian data sources, catalogue of other data catalogues. Older brother of Dateno search engine.
- Stat GIBDD — статистика аварийности Госавтоинспекции.
- To Be Precise — data collected by the authors of the project about social problems in Russia.
Moscow data
- Moscow Open Data Portal — official Moscow open data, categorized by topics.
- Integrated Data Warehouse — official Moscow open data, categorized by agency.
- PROdvizhenie — Moscow taxi, carsharing and e-scooter data. Not machine-readable, but daily updated hence useful.
- Map of road traffic accidents.
- Tinkoff Compass — aggregated spendings of Moscow residents, tutorial.
- Yandex Geoanalytics — data on population profile, pedestrian flows and car traffic. The car traffic figures are underestimated, but a better one is not publicly available, so it is acceptable to rely on this source.
Data of particular countries
Many countries have their own open data sites. They usually have gov in the domain (i.e. in the link address), e.g. https://www.data.gov.gr/ - Greek open data.
Sources of Inspiration
My absolute favourite — impressive narrative scrollytelling:
Computational linguistics (text analysis) as method
Impressive line charts
Impressive dots
Awesome data teams
- Millenials Are Screwed — Huffington Post
Computational linguistics (text analysis) as method
- Beautiful in English — Google News Lab + Visual Cinnamon — with VPN
- Literary Constellations — Nicholas Rougeux
- Why do cats and dogs...? — Nadieh Bremer + Google Trends
- What Do People Ask Most About Relationships In China? — ThePaper.cn (use Google Translate plugin)
Impressive line charts
- Going gray — Reuters Graphics
- How Americans Die — Bloomberg
Impressive dots
- Pompeii: 1) Visual story, 2) Landing page — Mohamad Waked
- The Fallen of World War II — Neil Halloran
- American DNA — MTI Senseable City Lab
Awesome data teams
Fun format
- Young Adults and the Future of News — Pew Research Center
- The Hidden Risks of Sesame Allergies — The Pudding
- Harry Potter fanfiction's favourite ships — Shirley Wu
- Sugar quiz: How much sugar is in our food? — ABC
Awesome data teams
- How do we compare? — Lindsey Poulter
- Atlas of Sustainable Development Goals 2023 — World Bank
IDK which category, but still
- History of Infographics — Info We Trust
- Women's Pockets are Inferior — The Pudding
- Correlation between trending search queries "Can't smell" and COVID outbreaks: The New York Times, Яндекс
- Traditional Chinese Color Libraries Browser — Zhejiang University
- Elections in the USA — RBC
Data Journalism & Visual Art Awards
Long-form articles are not always needed
Sometimes one picture is enough
Infographics are not always needed
Sometimes text is more beneficial. Experiment with the delivery of the message
Home Assignment
Deadline: 22 January, 18:00.
Grading criteria on the second sheet in the table
Grading criteria on the second sheet in the table
Find people you would like to work with in a group.
Create groups of 2-3 people and fill out this table in your homework folder using the example below.
1. Your names.
2. General theme of your project, may sound like scientific fields.
3. Specific wording of your project topic. It should answer traditional W-questions.
4. How come? — How have you come to this idea? What was first: idea, data, or have you used some mix? See "Idea + Data Strategies" block for more info.
5. The goal of your article & the main hypothesis or research questions:
6. Find 5 different data sources and datasets that are relevant to your project topic. Try to find different data formats for you to practice, but it's not necessary — however, use only the formats that we have discussed in class/tutorials.
7. Explain, why you chose these exact datasets and what kind of conclusions you want to retrieve from the analysis.
🌻 Infographics (ready-made picture) are not data! Look for primary sources, look for systematized object-by-object values.
🌻 Don't use Synthetic data! Look for data with a clear, understandable methodology and/or from a trusted source.
Create groups of 2-3 people and fill out this table in your homework folder using the example below.
1. Your names.
2. General theme of your project, may sound like scientific fields.
3. Specific wording of your project topic. It should answer traditional W-questions.
- Sample topic: Gender Inequality in World Literature from Biblical Times to the Present
- What? — Gender Inequality
- Which scientific field? — Literature
- Where? — In the whole world (you may also use only one country/ city etc.)
- When? — from Biblical Times to the Present
4. How come? — How have you come to this idea? What was first: idea, data, or have you used some mix? See "Idea + Data Strategies" block for more info.
5. The goal of your article & the main hypothesis or research questions:
- Why is it worth writing?
- Who needs this article?
- What kind of results do you expect?
- What do you want to find out?
6. Find 5 different data sources and datasets that are relevant to your project topic. Try to find different data formats for you to practice, but it's not necessary — however, use only the formats that we have discussed in class/tutorials.
- Link to Data Source — link to the general domen, where the data can be found.
- Example: https://data.mos.ru/
- Dataset Link — a link to a specific file with data. It can have a .csv, .xlsx, .pdf or something similar at the end of the URL. If it is not a file but a web table, the link should lead directly to the table.
- Example: https://data.mos.ru/opendata/62321?isDynamic=false
7. Explain, why you chose these exact datasets and what kind of conclusions you want to retrieve from the analysis.
🌻 Infographics (ready-made picture) are not data! Look for primary sources, look for systematized object-by-object values.
🌻 Don't use Synthetic data! Look for data with a clear, understandable methodology and/or from a trusted source.
