Content Area 6: Data

Rate each point. Be honest: 'Secure' means you could answer an exam question on it now without notes.

Sources of data (6.1.2)

Ethical data practices and the value of data (6.1.3)

Ethical practice means collecting only what is needed, being transparent, getting consent, keeping data secure, avoiding bias, and following data protection law. The value of data is judged by:

How organisations use data (6.1.4)

Data

Raw facts and figures with no context, e.g. 18, 22, 25.

Information

Data that has been processed and given context so it has meaning, e.g. 'Temperatures in Leeds this week: 18–25°C'.

Knowledge

Understanding gained from information that can be applied to make decisions, e.g. 'Ice cream sales rise above 20°C, so order more stock'.

Feedback loop (AI)

When an AI system's outputs influence the data it is later trained on, which can reinforce errors or bias.

Veracity

How accurate and trustworthy data is.

Methods of transforming data (6.2.1)

Data taxonomy (6.3)

Properties of qualitative data (6.3.5): it is stored and retrieved only as a single object (a photo or a review is saved and loaded whole; you can't query part of it like a database field), and it can be codified into structured data, e.g. tagging reviews as positive/negative, or rating comments on a 1–5 scale, so it can be analysed.

Practise: discrete, continuous or categorical?

Data types (6.4)

Quantitative data

Data that can be measured or counted as numbers.

Qualitative data

Descriptive data such as opinions, text, images or video.

Structured data

Data organised in a fixed format, such as rows and columns in a table.

Unstructured data

Data with no predefined format, such as emails, images and audio.

Discrete data

Numeric data that can only take specific values, usually counted, e.g. number of tickets.

Continuous data

Numeric data that can take any value in a range, usually measured, e.g. height.

Categorical data

Data that falls into a limited set of groups, e.g. T-shirt size S/M/L.

Real

A data type for numbers with a fractional part (float in Python).

Character

A data type holding a single letter, digit or symbol.

Blob

Binary large object: a data type for storing large binary data such as images, audio or files in a database.

JSON and CSV in Python

import json
import csv

order = {"id": 1042, "customer": "Asha",
         "items": ["latte", "scone"], "paid": True}

text = json.dumps(order)              # Python dictionary -> JSON string
print(text)
back = json.loads(text)               # JSON string -> dictionary
print(back["items"][1])

with open("orders.csv", "w", newline="") as f:
    writer = csv.writer(f)
    writer.writerow(["order_id", "customer", "total"])
    writer.writerow([1042, "Asha Patel", 5.40])

with open("orders.csv") as f:
    for row in csv.reader(f):
        print(row)

{"id": 1042, "customer": "Asha", "items": ["latte", "scone"], "paid": true}
scone
['order_id', 'customer', 'total']
['1042', 'Asha Patel', '5.4']

Notice the CSV reader returns every field as a string: CSV has no data types, so values must be converted after reading.

The same order in XML

<order id="1042">
  <customer>Asha Patel</customer>
  <item>latte</item>
  <item>croissant</item>
  <paid>true</paid>
</order>

Try it: is this valid JSON?

Edit the text, then check it. Try removing a quote mark or adding a comma after the last item.

ASCII and UTF-8

print(ord("A"), "A".encode("ascii"))
print(len("café".encode("utf-8")), "bytes in UTF-8")
try:
    "café".encode("ascii")
except UnicodeEncodeError:
    print("é cannot be stored in ASCII")

65 b'A'
5 bytes in UTF-8
é cannot be stored in ASCII

Structures for storing data (6.6)

Interrelationship (6.6.4): the storage structure affects how easily data can be transformed. Data scattered across separate files must be merged before analysis; hierarchical data may need flattening into rows before it can go in a CSV or spreadsheet.

JSON

JavaScript Object Notation: a lightweight text format storing data as key–value pairs and arrays, widely used by web APIs.

CSV

Comma-separated values: a plain text format where each line is a record and fields are separated by commas.

XML

Extensible Markup Language: a text format that uses custom tags to describe and structure data.

ASCII

A character set using 7 bits to represent 128 characters (English letters, digits, symbols, control codes).

UTF-8

A variable-length Unicode encoding (1–4 bytes per character) that can represent characters from every language and emoji, and is backwards compatible with ASCII.

Metadata

Data about data, e.g. a photo's date, location, file size and camera model.

Big Data is data that is too large, fast or varied for traditional tools to handle. It has multiple dimensions, the six Vs:

Data quality assurance methods (6.7.4)

Factors that affect maintenance (6.7.5): time (cleaning and updating data takes staff time), skills (specialists such as data engineers are needed for large data sets), cost (storage, software, staff, training).

Big Data

Data sets so large, fast-moving or varied that traditional tools can't process them; described by the six Vs.

Verification

Checking data has been entered or transferred accurately, e.g. double entry or proofreading.

Data integrity

Data remains accurate, complete and consistent throughout its life.

Data redundancy

The same data stored in more than one place. Unplanned redundancy causes inconsistency; planned redundancy (backups) improves resilience.

Data wrangling (6.8.1–6.8.2)

Data wrangling transforms raw data from different sources into a clean, consistent format ready for analysis. It is used before analysis, when combining data sets, or importing data into a new system.

Core functions of a data system (6.8.3)

Data entry errors (6.8.4–6.8.8)

Factors that impact data entry (6.8.6): time needed to create the screens; expertise needed to create the screens; time needed to enter the data. Better screens (drop-downs, lookups, validation) reduce errors and entry time, but cost more time and expertise to build, so the choice depends on how much data will be entered and how costly errors are.

Data wrangling

Transforming raw data into a clean, structured format ready for analysis.

Transcription error

A mistake made when copying data, e.g. typing 'Smtih' for 'Smith' or a wrong digit.

Transposition error

Two characters swapped, e.g. 1243 instead of 1234.

Visualisation formats (6.9)

Choose based on the type of data (categories → bar chart; time series → line graph), the intended audience (directors → dashboard; public → infographic; analysts → tables) and the brief (what decision is it for?).

Data models (6.10)

Hierarchical model

Network model

Relational model

Data stored in tables (relations) of rows (records) and columns (fields), linked by keys.

Hierarchical model

Data organised as a tree: each child record has one parent.

Network model

Data organised as a graph: records can have many parents and many children (many-to-many).

Relational model

Data organised in tables linked by primary and foreign keys.

Primary key

A field that uniquely identifies each record in a table.

Foreign key

A field in one table that is the primary key of another table, creating a link.

Accessing data across platforms (6.11)

Data analysis tools (6.12)

Scale (6.12.2): a small business can analyse data in a spreadsheet; as volume and variety grow, a database and reporting tool are needed; with Big Data, warehouses, lakes and mining tools become necessary.

RBAC

Role-based access control: permissions given according to the user's role.

RuBAC

Rule-based access control: access decided by rules, e.g. time of day or location.

API

Application programming interface: a defined way for one program to request data or services from another.

Data warehouse

A central store of structured, cleaned data from many sources, organised for reporting and analysis.

Data lake

A store holding large amounts of raw data in any format until it is needed.

Data mart

A smaller subset of a data warehouse focused on one department or topic.

Data mining

Using algorithms to find hidden patterns and relationships in large data sets.

CRM

Customer relationship management: systems that store and analyse customer data and manage communications.

Flashcards

Every Quick recall definition on this page. Click the card to flip it; use the arrow keys to move.

Random questions on CA6 from the course question bank. Pick a topic or question type, answer in your head or on paper, then check. Your score is saved on this device.

Allow about 90 minutes.

My corrections: what I need to go back over