This project had two parts:

Part A was focused on comparing programming languages and their usage. I chose to compare C and Python because I am familiar with both and know they have extreme differences. I also chose to demonstrate these differences using a data processing application - something that is very easy to do in Python using Pandas and only took a couple of hours in total. In C, the same exercise took around three times longer because of no libraries that would be capable of doing the same job. It also took a lot of manual work when parsing the CSV file, something that was only one line in Python suing Pandas, tunrned out to be a whole file of code in C.

Link to a GitHub project: https://github.com/lenasolarova/accident-analysis

Part B was delivering a 15-minute presentation on a topic regarding new emerging technologie, their practical usage and legal frameworks. I chose AI in people management as it is something very topical for most of us, yet underrepresented in many scientific papers. I focused on defining AI from both technical and legal perspective, using plenty of examples and finally demonstrating issues on a case study of AI usage in hiring in Amazon.

Skills learned: Python programming, basic algorithm design, and effective problem-solving.

Large-scale data collection for IoT (focused on health monitoring)

The initial post titled was focused on IoT devices in healthcare and specifically in health monitoring. Highlighting the opportunities such monitoring provides not only when it comes to individual health but also focusing on the bigger picture and mass life-saving capabilities.

The following sections were diving deeper into the challenges IoT devices inevitably have such as the physical limitations of collecting vast amount of data or processing the data effectively - working with noise, missing values or unexpected data. All of which in the field of healthcare affects not only the daily work of data scientists but possibly also the health of patients.

This field of study is also highly topical from the personal data and sensitive data perspective as most of us are allowing our devices to collect wide range of data even without any special devices (f.e. our phones are capable of recording our location in real time -> speed or pace -> possibly assessing our health using algorithms).

Feedback on my discussion post

My post concerning IoT in health monitoring was received positively in all purely technical areas yet people were rightfully critical towards the lack of deeper examination of the regulatory framework concerning biometric data and personal data as a whole. This is obviously crucial in any field which has the capacity to collect, process and store data as sensitive as biometric ones or even generic personal data. It is necessary to always keep in mind that GDPR and potentially national laws provide a quality legal framework for data collected, processed and stored in and out of the EU.

Note: This was also a topic of the discussions in units 8 to 10 as is further dissected there.

My feedback on others' discussion posts

I provided feedbak on multiple posts as well. The other students' posts ranged from IoT in Smart factories which is a very intriguing space where IoT devices can be used to limit the labour costs and at the same time protect the workers by employing smart sensors. It is also a specific field where perfectly structured and clean data is practically non existent and the pressure on the data cleaning and processing is high. I pointed out a similar deficit as was pointed out in my post and that is the lack of an regularoty framwork dissection.

Another posts were more generic and did not provide any specific examples or areas where IoT would be used, which I pointed out as a flaw.

Goal

The goal of this project was to teach us how to utilise bs4 and requests libraries in python to do very simple web scraping using the websites' HTML.

General idea

The general idea of the exercise was about choosing any website and using the appropriate libraries to scrape the website and find the term "data scientist". I decided to scrape a job portal (Czechia's biggest one after Indeed - jobs.cz) and specifically the seach results for "data scientist".

Challenges and Implementation

I initially chose to scrape indeed.com but I came across many issues with the website returning error 403 (Access denied) even when I used a token since it recognised I was trying to web scrape it and was not a normal user. Instead of using indeed.com I decided to focus on jobs.cz which is a smaller more local website and the access was no issue then.

Implementation itself was very simple, the only issue was inspecting the job portal website's HTML to find out which elements contained the appropriate data to filter out. This also brings our one of the main challenges this solution would have on a larger scale and that's consistency across websites. While jobs.cz may have the data in h2 elements this is not the case for other websites thus confirming the data collection and filtering process is usually one of the most time consuming tasks of a data scientist.

The result is saved into JSON.

Link to a GitHub project: https://github.com/lenasolarova/web_scraping_jobs

The general idea

The general point of this team project was to choose a sector of our interest and prepare a database design for this company considering all requirements and limitations, providing a quality design we will be able to actually build and test at a later stage individually.

The team

The team project started around second week of the course by assembling our team of four people. Everyone turned out to be smart and kind so the team work was fairly straightforward with minor hiccups just like any team work has.

The process

We had decided to work on a database design for a large B2B retailer as two people from the team had experience in the field, me included. We sechduled weekly meetings and on our first meeting we simply talked about what we wanted to work on and general idea about timelines and decided to work on a draft before the second meeting. I had drafted a document covering every area of the assignment and other team members helped primarily review and add ideas at this stage. At the second meeting we divided the work properly into four sections and I got to work on an area that I was not extremely familiar with but excited to learn and that was the technical database design, diagrams and data types and attributes of each table. By the following week we all had a decent draft of our section and all that remained was formatting, corrections and making sure we managed to fit into our word count which was rather stressful but we managed it really well in the end two days before the deadline.

Database Normalisation

Original Table (Unnormalised)

First Normal Form (1NF)

Actions taken:

Identified primary key (Student Number), Removed repeating groups, Ensured all values are atomic, Moved course-related data into a separate table, Created a junction table for the many-to-many relationship

STUDENT

COURSE

STUDENT_COURSE

Second normal form (2NF)

No composite primary keys are used in the STUDENT or COURSE tables, therefore, there are no partial dependencies so the database already satisfies Second Normal Form

Third normal form (3NF)

Actions taken:

Removed transitive dependencies, Replaced string identifiers with surrogate keys, Separated repeating teacher and exam board data into their own tables

TEACHER

BOARD

COURSE 3NF

STUDENT 3NF

Database Build

After the initial exercise of noramizing the table into 3NF, it was time to start building and testing it in real life. I chose to use PostgreSQL as a databse of choice as I never worked with it directly.

This phase consisted first of learning how to even start the PostgreSQL server, which turned out to be a few simple commands

sudo systemctl start postgresql

sudo systemctl status postgresql
          

And then I needed to initialize the database for the first time:

psql -U postgres -d postgres

CREATE DATABASE university;
          

And then it was time to fill the database with data and write the actual tests for the database

The test suite is based on pytest.

pip install -r requirements.txt

pytest -v

Additional Remarks

The assignment requires testing of referential integrity, which means that every foreign key must refer to an existing primary key value in the linked table. Simply put, if a course references a teacher with teacher_id = T100, then a teacher with teacher_id = T100 must exist in the TEACHER table.

PostgreSQL automatically enforces referential integrity through foreign key constraints. For example, the following constraint ensures that each course references a valid teacher:

FOREIGN KEY (teacher_id) REFERENCES teacher(teacher_id)

Referential integrity is also explicitly tested in the automated test suite using pytest, specifically in the test_referential_integrity test case.

Link to a GitHub project: https://github.com/lenasolarova/accident-analysis

GDPR vs. Personal Data Processing Act (Czechia)

The original post focused on the principle of personal data security and compared how this principle is addressed in the General Data Protection Regulation (GDPR) and in the Czech Personal Data Processing Act (Zákon č. 110/2019 Sb). The introduction briefly outlined the challenges lawmakers face in protecting personal data in an increasingly digital and interconnected world where data frequently crosses national borders.

The following sections explored how GDPR defines the security of personal data through the requirement of appropriate technical and organisational measures. These include practices such as pseudonymisation, encryption, system redundancy, and regular testing, all aimed at preventing unauthorised access, data loss, or damage while taking into account proportionality and risk.

In contrast, the Czech Personal Data Processing Act was discussed mainly as a supplementary framework. Rather than redefining security obligations, it builds upon GDPR by addressing specific processing contexts, such as law enforcement or national security, while relying on GDPR as the primary and directly applicable source of security requirements.

Overall, my post concluded that both legal frameworks emphasise the importance of protecting personal data, with GDPR setting the core standards and Czech legislation refining their application in local and sector-specific scenarios. The discussion highlighted that effective data protection is not about absolute security, but about demonstrating that appropriate and proportionate measures are in place.

Link to the post: https://www.my-course.co.uk/mod/forum/discuss.php?d=339998

API Security Requirements Specification: GitHub API vs. GitLab API

There is only a small number of APIs used for various and extensive purposes, such as those provided by the two largest Git hosting services – GitHub and GitLab (Ghodke and Chavan, 2024) – which is exactly why they should follow best practices when it comes to security. This report examines both APIs using a real-life example of a Python-based application that scrapes merged pull and merge requests and stores the results in JSON files for further visualisation in Grafana (GitHub, no date).

System Overview

The application is executed via a CI/CD pipeline triggered on a schedule, manually, or on repository commits. The pipeline runs a Python script that queries the GitHub and GitLab APIs over HTTPS, retrieves metadata related to merged pull and merge requests in JSON format, and stores the processed data in version-controlled JSON files within a GitHub repository. Grafana then reads these JSON files to visualise the collected metrics.

Threat Risk Identification

The most common threats related to API usage originate from multiple sources. APIs typically require authentication mechanisms such as access tokens or credentials and enforce rate limits to prevent excessive or abusive usage. Client-side risks include token leakage, insecure token storage, or the use of overly broad access scopes. Additional risks may arise on the provider side, including the unintended exposure of data beyond the expected scope or the inclusion of sensitive information within API responses (Basak and Tiwari, 2025, pp. 19–20).

Security Requirements

API Comparison

Both the GitHub and GitLab APIs require personal access tokens to authenticate requests. When the scraping tool is executed within GitHub Actions, a token is provided automatically by the execution environment. In contrast, access to the GitLab API requires an explicitly configured token, even when executed within GitLab CI/CD. Both platforms impose rate limits that vary depending on request type, thereby reducing the risk of automated or bot-driven scraping (GitHub Docs, no date; GitLab Docs, no date).

Logging is an important security mechanism for detecting abnormal behaviour and potential breaches. GitLab provides user-accessible logging with multiple log levels ranging from DEBUG to UNKNOWN (GitLab Docs, no date). GitHub, however, offers limited direct visibility into request-level logging for API consumers.

Conclusion

Both the GitHub and GitLab APIs fulfil essential security requirements when used correctly. By enforcing authenticated access, scoped permissions, encrypted transport, rate limiting, and secure secret management through CI/CD pipelines, the identified risks can be effectively mitigated. This analysis demonstrates that a Python-based integration using JSON data formats can be securely implemented when appropriate security requirements are clearly defined and consistently applied.

References

Basak, A. and Tiwari, D. (2025) API Security Risks and Resilience in Financial Institutions. Master's thesis. Laurea University of Applied Sciences. Available at: https://www.theseus.fi/bitstream/handle/10024/883344/Basak_Tiwari.pdf (Accessed: 30 December 2025).

Ghodke, G. M. and Chavan, T. (2024) ‘An Overview of Git’, International Journal of Scientific Research in Modern Science and Technology, 3(6). Available at: https://doi.org/10.59828/ijsrmst.v3i6.216

GitHub Docs (no date) Rate limits for the REST API. Available at: https://docs.github.com/en/

Database proposal

This database design proposal focused on addressing the data management needs of ElectroSpares, a B2B wholesale supplier of office equipment to corporate clients. The proposal focused on not only fulfilling the technical needs of such client when it comes to proper logical design, but also when it comes to data security or efficiency.

Evaluation against the database build

The database build structurally follows and builds on top the proposal from unit 6 focusing more on business needs of such comapny. The final project therefore includes not only the properly built database tested with artificial data to ensure proper data handling but also focuses on the analytical side which was only lightly touched on in the proposal.

The biggest improvement personally between the initial proposal and the final build was gaining vast amount of knowdledge and context in the module via reading or working on smaller activities. These activities made it possible for me to consider more than was possible in the initial proposal such as the business intelligence needs or proper database building tools which I was not aware of before. Based on that knowdledge I was able to implement various parts of the data pipeline instead of only focusing on the database itself. I managed to create the tables, connect them to business intelligence tools and create visualisations. All while experimenting with different tools and searching for the ideal ones for the task. This research was again one of the most memorable moments of the assignment as it made me connect all the things I learned so far with existing technologies such as considering the usage of NoSQL databases versus SQL ones for business intelligence purposes or researching how S3 fits into a properly designed data piepleine or even how to correctly connect a database and visualisstion tools without causing strain on the main database system.

The initial proposal was a vital piece of work for the build since our team spent a lot of effort on it. This made me realise that it is important to not underestimate the value of every single assignemnet. Every assignemnt well done is a small step towards being a better data science student and professional. This is not to say that every assignemnt has gone to plan and that was certainly true for my database build as I at times missed the team who would come up with ideas or correct assumptions which is something I value immensly.

What

This module covered the full spectrum of machine learning, from the mathematical foundations like correlation and regression all the way to deep learning with CNNs and transfer learning. Over the course of twelve units, I went from doing basic EDA on the Auto-MPG dataset to building a full pneumonia detection system comparing an SVM against a ResNet18 CNN for my final project. The module also had a team project in Unit 6 where we analysed Airbnb pricing data using anomaly detection and self-supervised learning, and throughout there was a strong emphasis on ethics, explainability and the real-world implications of deploying ML systems.

So What

Machine learning algorithms. What surprised me the most was how many different approaches exist for what seem like similar problems. In the early units I learned about linear regression and SVMs, then moved to unsupervised methods like K-Means clustering, and finally to neural networks and CNNs. Each technique has its strengths, SVMs are fast and work well on smaller datasets while CNNs can automatically learn features from images but need more data and computing power. I think the most valuable thing I took away is that choosing the right algorithm is not about picking the most complex one but about understanding the trade-offs. In my final project, the SVM actually had slightly higher recall than ResNet18 (99.7% vs 99.5%), but the CNN won overall because of better accuracy and the ability to provide visual explanations with Grad-CAM.

Team contributions. The team project was honestly a mixed experience. We were assigned to Track 3 (Advanced ML) and had to analyse Airbnb data using self-supervised learning and Neural Architecture Search. Forming the team was slow because some members took three or four weeks to respond to messages, and even after the team was assembled not everyone contributed equally. Paul was the team leader and by far the most active contributor, he handled the preprocessing, NAS optimisation and most of the organisational work. I took on the exploratory data analysis, creating visualisations of pricing patterns across neighbourhoods, room types and seasons. I also helped clean the data and convert an R preprocessing script into a Jupyter notebook so it was consistent with the rest of the project. Other team members contributed to the anomaly detection model and evaluation but their involvement was less consistent.

Working in a team where some people were less engaged was frustrating but also taught me something important. At work I am used to a team where everyone pulls their weight, so dealing with uneven contribution in a university setting was a different challenge. I learned that you cannot always rely on everyone doing their part and sometimes you just have to step up and fill the gaps rather than waiting around. Paul and I ended up doing most of the work between us, and while that was not ideal it did mean we both got a much deeper understanding of the full pipeline.

Ethical and professional awareness. Before this module I thought of ethics in AI as something abstract, bias and fairness were topics I had heard about but never really connected to my own work. That changed significantly especially during the final project. When I saw that my pneumonia detection model had 128 false positives (healthy patients flagged as having pneumonia) vs only 2 false negatives (missed pneumonia cases), I realised how the choice of metrics directly affects real people. Reporting 79% accuracy sounds fine but it hides the fact that over half of normal patients would get unnecessary follow-ups. In medical AI that matters enormously. The Grad-CAM and LIME explainability work also made me appreciate why black-box models are problematic in healthcare, if a doctor cannot see why the model flagged a case, they cannot trust it. The EU AI Act (2024) classifying medical AI as high-risk suddenly made a lot of practical sense to me.

Now What

This module has had a clear impact on how I think about my professional work. At work I deal with data pipelines and understanding how models consume data and what happens when data quality is poor gives me a much better perspective on why the pipelines I maintain matter. I now understand that the preprocessing and data quality work we do is not just a technical requirement, it directly affects model reliability and by extension the decisions that depend on those models.

On a personal level I gained confidence in Python libraries I had not used before (PyTorch, scikit-learn, LIME) and in working with image data which was completely new to me. The final project pushed me to learn transfer learning, Grad-CAM, and cross-validation, all of which I can see applying in future modules or in personal projects. I also developed a much stronger appreciation for explainability, not as an academic box-ticking exercise but as a genuine professional responsibility.

One thing I did find a bit disappointing is that most of the work in this module stayed within Jupyter notebooks with no real deployments. The final project discussed deployment on AWS SageMaker but it remained theoretical, and the same was true for the other exercises. I understand that actual cloud deployment would require time, infrastructure and costs that are not realistic within a 12-week module, but I would have liked at least a small hands-on exercise with exporting a model or setting up an inference endpoint. That said, this gap has motivated me to explore deployment tools like SageMaker or ONNX on my own since it feels like the missing piece between building a model and actually using it in the real world.

Going forward I also want to explore self-supervised learning more since the team project gave me a taste of it and it seems like the direction the field is moving. Overall this module took me from knowing the basics of Python and data to feeling like I could actually build, evaluate and critically assess a real ML system, and that is something I did not expect when I started.

References

European Parliament and Council of the European Union (2024) Regulation (EU) 2024/1689 — The Artificial Intelligence Act. Official Journal of the European Union.

Rolfe, G., Freshwater, D. and Jasper, M. (2001) Critical reflection in nursing and the helping professions: a user's guide. Basingstoke: Palgrave Macmillan.

The Assignment

This was a team project where we had to analyse the Airbnb NYC 2019 dataset using our assigned machine learning track. We got Track 3 (Advanced ML) which meant using self-supervised learning for anomaly detection and Neural Architecture Search (NAS) for model optimisation. The goal was to identify listings with unusual pricing patterns — either significantly overpriced or underpriced compared to similar properties.

The Team

Getting the team together was honestly the hardest part. Some members took three to four weeks to even respond to the initial messages about forming the group. Once we were assembled we had six people but not everyone contributed equally. Paul was the team leader and the most consistent contributor — he handled data preprocessing, NAS optimisation and kept the project moving forward. I took on the exploratory data analysis and also helped with data cleaning. The other members contributed to the anomaly detection model and evaluation but with varying levels of involvement.

My Contribution

I was responsible for the Exploratory Data Analysis (Section 3 of the project). I created visualisations covering:

• Price distribution across all listings (right-skewed, most under $200)
• Price by neighbourhood (Manhattan highest, Bronx lowest)
• Average price by room type (entire homes most expensive, shared rooms cheapest)
• Seasonal pricing trends by month
• Relationship between reviews and price
• Availability distribution across the year
• Correlation heatmap of all numeric features

I also converted Paul's initial R preprocessing script into a Jupyter notebook so it was consistent with the rest of the project, and I cleaned the dataset by removing listings with zero prices.

Key Findings

The EDA revealed that most Airbnb listings in NYC are priced under $200, with entire homes being roughly twice as expensive as private rooms. Manhattan had the highest median prices and the most outliers. The anomaly detection model (autoencoder) successfully identified listings that were priced far above or below what their features would suggest, which could be useful for Airbnb's pricing strategy team to flag suspicious or mispriced listings.

Link to the team GitHub repository: https://github.com/Mr-Ratman/ML_team_project

Self-Supervised Learning

One thing that really stood out to me during the module was how much work goes into labelling data. For my final project I used 5,856 chest X-rays that had to be verified by two doctors each, which is a huge amount of effort. So I found self-supervised learning (SSL) really interesting because it lets models learn from unlabelled data instead, using tricks like predicting masked parts of images or contrastive learning. This seems especially useful in medical imaging where getting labelled data is expensive and doctors are busy enough as it is. I came across models like DINO and MAE that apparently can match or even beat supervised pretraining, which could make transfer learning even better than the ImageNet approach I used with ResNet18.

Neural Architecture Search (NAS)

For my project I picked ResNet18 because I knew it was a solid architecture that works well without needing too much computing power. But there is this concept called Neural Architecture Search that basically automates the whole process of choosing a model architecture. It tries out different combinations of layers, filter sizes and connections to find the best one for your specific task and data.

Advantages: It can find architectures that are actually better than what humans design, which is pretty cool. For example EfficientNet was discovered through NAS and it gets better accuracy than ResNet with fewer parameters.

Challenges: The downside is that it is extremely expensive to run. Some of the early NAS papers used thousands of GPU hours which raises questions about energy usage and carbon footprint. It also means only big companies with lots of resources can really afford to do this.

Edge AI

In my final project I talked about running the pneumonia detection model on hospital servers instead of in the cloud. This is basically Edge AI, where you deploy models on devices that are close to where the data comes from.

Applications:

• Medical devices: You could do real-time X-ray screening in rural clinics that do not have reliable internet, which is exactly what my pneumonia model could be used for
• Autonomous vehicles: Decisions like braking or swerving cannot wait for a round-trip to the cloud so the model has to run on the device itself
• Smart cities: Things like traffic management or air quality monitoring using local sensor data

There are compression techniques like quantisation, pruning and knowledge distillation that make this possible. My ResNet18 model could be converted to ONNX or TorchScript and quantised to INT8 to run on smaller devices without losing much accuracy.

Ethics, Scalability & Sustainability

AI is developing so fast that it brings a lot of challenges that are all connected:

• Ethics: The more capable models get the more they can be misused. The EU AI Act (2024) is a good start, it classifies medical AI as high-risk and requires proper assessments. But regulation always seems to be a step behind the technology so it needs to keep evolving.
• Scalability: When models scale to billions of parameters like GPT or LLaMA it creates a situation where only a few big companies have the resources to build them. I think open-source models and smaller task-specific ones like ResNet18 are a more accessible way forward.
• Sustainability: Training large models uses a lot of energy. I read that Strubell et al. (2019) estimated training one large transformer produces as much CO2 as five cars over their entire lifetimes which is honestly shocking. Transfer learning, where you reuse pretrained models instead of training from scratch, is not only more efficient but also better for the environment.

Reflection

Looking back at the whole module, the things I worked on in my final project like transfer learning, explainability, bias analysis and deployment planning turned out to be really practical skills, not just theory for an assignment. I think the future of ML is not really about making bigger and bigger models but about making them more efficient, understandable and fair. That is the part I found most interesting and I could see how it connects to real work.

Overview

This exercise was about understanding how changing model parameters affects performance metrics, specifically AUC (Area Under the ROC Curve) and R² (coefficient of determination). I found it really helpful to actually see how tweaking things changes the results, it made me realise that evaluating a model is not just about one number.

Key Observations

AUC (classification):

• AUC basically measures how well a model can tell classes apart. A score of 1.0 means perfect separation and 0.5 is just random guessing.
• I noticed that making the model more complex (like adding more trees in a random forest) improved AUC up to a point but then it started overfitting and the score on validation data stopped improving or even got worse.
• What I found useful is that AUC does not depend on a specific threshold so it works better than accuracy for imbalanced datasets. This was really relevant for my final project where the pneumonia dataset had a 2.89:1 class imbalance.

R² (regression):

• R² tells you how much of the variance in the target variable the model explains. 1.0 is perfect and 0.0 means the model is no better than just predicting the average.
• I also learned that R² can be misleading because adding irrelevant features can inflate it artificially since the model just memorises noise. That is why adjusted R² or cross-validated R² are more reliable.

Ethical Dimension

Something that stuck with me from this exercise is that the choice of metrics has real ethical consequences. If you only report accuracy on an imbalanced dataset you can hide some serious problems. A model with 90% accuracy might actually miss every positive case if the negative class is much bigger. In medical AI that could mean missed diagnoses, in criminal justice it could mean wrongful classifications. In my final project I deliberately focused on recall over accuracy because missing a pneumonia case is far more dangerous than a false alarm.

CNN Exercise

I ran the CNN object recognition notebook using the CIFAR-10 dataset which has 60,000 colour images across 10 classes like airplane, car, bird, cat, dog, frog and so on. After training the model I tested it by changing the prediction index from x_test[16] to different values to see if it would get the predictions right.

It got most images right but sometimes confused similar looking categories like cat vs dog or automobile vs truck. This made it clear to me that CNNs learn from pixel patterns and can struggle when classes look similar, which is something that could be a real problem in more serious applications.

Ethical & Social Implications of CNN Technology

Wall (2019) brings up some important concerns about CNN-based recognition systems being used in the real world. After working with CNNs in both this exercise and my final project on pneumonia detection, a few ethical issues stood out to me:

1. Surveillance and privacy: CNN-powered facial recognition is being used more and more in public spaces, often without people knowing about it. The same kind of architecture that classifies pictures of trucks and cats can be trained to identify people, which raises questions about privacy and anonymity. Unlike regular CCTV, CNN systems can automatically identify, track and profile people at a large scale.

2. Bias and discrimination: CNNs pick up whatever biases exist in their training data. Facial recognition systems have been shown to perform much worse on darker-skinned individuals and women (Buolamwini and Gebru, 2018). If a model is mostly trained on images from one demographic it will not work well for others, and in law enforcement that could lead to wrongful arrests which is really concerning.

3. Accountability gap: When a CNN makes a serious mistake like misidentifying a suspect or missing a tumour, it is not clear who is responsible. Is it the developer, the company using it, or the algorithm itself? Deep learning models are often called "black boxes" because it is hard to understand how they make decisions. This is why explainability tools like Grad-CAM, which I used in my final project, are not just technically interesting but really necessary.

4. Dual-use nature: The same CNN architecture that helps with medical imaging could also be used for authoritarian surveillance. This is something I think ML people need to think about seriously, not just whether a model works but how it could be misused.

Reference

Wall, M. (2019) 'The good, the bad and the ugly uses of machine learning in business', BBC News. Available at: https://www.bbc.co.uk/news/business-50365609

Buolamwini, J. and Gebru, T. (2018) 'Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification', Proceedings of Machine Learning Research, 81, pp. 77–91.

Overview

This exercise was about gradient descent which is basically the optimisation algorithm behind pretty much all modern machine learning. I ran the tutorial and played around with the iteration count and learning rate to see how they affect the training process.

Key Observations

Effect of learning rate:

• Too small (e.g. 0.001) — the cost goes down very slowly and needs a lot of iterations to converge. It gets there eventually but wastes computing time.
• Just right (e.g. 0.01) — the cost decreases steadily and converges in a reasonable number of iterations. This is the sweet spot.
• Too large (e.g. 1.0) — the cost starts jumping around or even gets worse. The algorithm basically overshoots the minimum and bounces around it or flies off completely.

Effect of iterations:

• More iterations help reduce the cost but you get diminishing returns since most of the improvement happens early on.
• Too few iterations and the model has not converged yet which means underfitting.
• This is why early stopping, which I used in my final project with ResNet18, is so useful. It automatically stops training when more iterations are not helping anymore.

Connection to the Module

Gradient descent is basically the engine behind all neural network training including the ResNet18 model I used in my final project. The Adam optimiser I chose is a fancier version that adjusts the learning rate for each parameter and includes momentum, but the core idea of stepping downhill on the cost surface is the same. Understanding this foundation helped me make a lot more sense of the training curves in the final project where I could see the loss decreasing over epochs and the learning rate being adjusted by the scheduler.

Reference

Mayo, M. (2017) A Gentle Introduction to the Gradient Descent Procedure for Machine Learning. KDnuggets.

Task 1: Model Comparison in Practice

I read a paper by Yu et al. (2024) that compares a CNN against four traditional ML models (Random Forest, SVM, MLP, Decision Tree) for predicting supply chain distribution costs using about 180,000 data points.

What problem is being solved?
It is about predicting distribution costs in supply chain management, which is a regression task. Getting this right helps with resource allocation and operational efficiency.

Quantitative comparison:
The paper compares models side by side using R² and RMSE on the same test set. The CNN got an R² of 0.953 with RMSE of 0.528 which beat all the traditional models. This is similar to what I did in my final project where I compared SVM and ResNet18 using accuracy, precision, recall, F1 and AUC-ROC on the same chest X-ray test set.

Trade-offs:

Task 2: Explainability and Trust in AI Systems

Since my assignment track was Computer Vision focused on medical imaging, the idea is that ML and DL models help radiologists detect pneumonia from chest X-rays. Basically the model does the initial screening and flags suspicious cases for the doctor to look at.

Concerns without explainability: If a model flags someone as having pneumonia but cannot explain why, doctors have no way to verify if the reasoning makes sense. They are not going to change their clinical judgement based on something they cannot understand. And if a model misses a case, without explainability tools there is no way to figure out what went wrong.

How XAI techniques help:

• Grad-CAM — creates heatmaps showing which parts of the X-ray the CNN was looking at. In my final project Grad-CAM showed the model was focusing on the lung opacity regions which is what you would expect clinically.
• LIME — breaks the image into superpixels and figures out which parts support or go against the prediction. I used this on the SVM model to check it was using clinically relevant features.
• SHAP — gives consistent feature attribution scores. This is especially useful for tabular data and recommender systems where you need to explain individual feature contributions.

As someone who works in development at Red Hat, I think it is important that any deployed model has explainability built in from the start, not added as an afterthought. Especially in medical AI this is becoming a regulatory requirement under things like the EU AI Act.

Task 3: Recommender System Exercise

I built a collaborative filtering recommender system using the MovieLens 100K dataset which has 100,000 ratings from 943 users on 1,682 movies. It was interesting to go through different approaches from simple to more complex:

• Global mean predictor — the simplest baseline, it just predicts the average rating for everything
• Bias model — takes into account that some users tend to rate higher and some movies are generally more liked
• Matrix Factorisation (MF) with SGD — this is the classic collaborative filtering approach where you break down the sparse user-item matrix into latent factor vectors and train it with stochastic gradient descent

For evaluation I used both rating prediction metrics (RMSE, MAE) and ranking metrics (Precision@K, Recall@K, NDCG@K). I also applied SHAP through a surrogate model to explain why certain user-movie predictions were high or low. For example it could show that a user's preference for action movies and a film's high average rating both contributed to a high predicted score, which I thought was quite neat.

References

Yu, X., Tang, L., Long, L. et al. (2024) 'Comparison of deep and conventional machine learning models for prediction of one supply chain management distribution cost', Scientific Reports, 14, 24195. Available at: https://doi.org/10.1038/s41598-024-75114-9

Mienye, I.D. and Swart, T.G. (2024) 'A comprehensive review of deep learning: architectures, recent advances, and applications', Information, 15(12), 755.

Molnar, C. (2023) Interpretable Machine Learning (2nd ed.). Available at: https://christophm.github.io/interpretable-ml-book/

Overview

This unit was about the building blocks of neural networks. We did three exercises that went from a single artificial neuron all the way to a multi-layer network which was a nice way to see how things build on each other.

Exercises Completed

1. Simple Perceptron — I implemented a single-neuron model that takes weighted inputs, adds them up and applies a step function to give a binary output. It is the simplest neural network you can have and it can only learn patterns where a straight line can separate the two classes.

2. Perceptron AND Operator — Here I trained a perceptron to learn the logical AND function which showed how the weights and bias get adjusted through learning. Since AND is linearly separable a single perceptron can learn it perfectly. What I found interesting was understanding why a single perceptron cannot learn XOR, which was actually the reason people developed multi-layer networks in the first place.

3. Multi-layer Perceptron (Sigmoid Function) — This extended the single perceptron into a multi-layer architecture with a sigmoid activation function. The sigmoid gives smooth continuous outputs between 0 and 1 instead of just 0 or 1, which is what makes gradient-based learning (backpropagation) possible. This is basically the foundation of deep learning and the same principle that the ResNet18 model in my final project uses, just scaled up to millions of parameters.

Key Takeaways

Going from a single perceptron to a multi-layer network really helped me understand a core ML concept: by adding layers and non-linear activation functions you allow the network to learn more complex patterns. A single perceptron can only draw straight boundaries but a multi-layer one can draw curves and capture non-linear relationships. That is why deep learning works so well for things like image recognition.

Link to the GitHub repository: https://github.com/lenasolarova/ML_module

Overview

In this exercise I applied K-Means clustering to three datasets that got progressively more complex. I compared the clusters the algorithm found against the actual labels where they existed, and also explored different ways to pick the right number of clusters.

Task A: Iris Dataset

I clustered 150 iris samples using four flower measurements with K=3. The algorithm did a great job separating setosa from the other species but there was some overlap between versicolor and virginica, which makes sense since they are genuinely quite similar botanically. The Adjusted Rand Index confirmed the clusters matched the real species labels well.

Task B: Wine Dataset

This one used 178 wines with 13 chemical properties. I learned that feature standardisation is really important here because without scaling, features like Proline (values around 500-1700) would completely dominate over features like Hue (values around 0.5-1.7) when calculating distances. After standardising, K=3 gave clusters that matched the three wine cultivars reasonably well. I also used PCA to project the 13 dimensions down to 2D so I could actually visualise it.

Task C: WeatherAUS Dataset

This dataset was different because it has no predefined labels so it was a truly unsupervised scenario. After cleaning up the data by removing non-numeric columns and handling missing values, I tested K=2 through K=6. I used the elbow method and silhouette scores to help pick K, and the scatter plots showed how more clusters create finer weather pattern groupings.

Key Takeaways

I learned that K-Means assumes spherical clusters of similar size which works fine for something like Iris but is a limitation with messier real-world data. Also feature standardisation is not optional, you really need it when features have different scales. Choosing K either comes from domain knowledge (like knowing there are 3 species) or from analytical methods like the elbow and silhouette when you do not have ground truth.

Link to the GitHub repository: https://github.com/lenasolarova/ML_module

Problem

The task was to take pathological test results for three people and calculate the Jaccard distance (which measures dissimilarity) for each pair.

Original Data

Binary Encoding

Gender gets excluded because it is a symmetric binary attribute where neither value is more significant than the other. For the rest of the attributes I encoded Y/P as 1 (positive/present) and N/A as 0 (negative/absent).

Jaccard Distance Formula

Jaccard coefficient J = f₁₁ / (f₁₀ + f₀₁ + f₁₁)
Jaccard distance d = 1 − J

The key thing here is that f₀₀ (where both are zero) gets excluded because for asymmetric binary attributes, the fact that two people both do not have something is not really informative.

Results

Interpretation

So Jack and Mary turned out to be the most similar pair with a distance of 0.33 since they both had positive results on Fever and Test-1. Jim and Mary were the most different at 0.75, only sharing Fever as a common positive. This exercise helped me see why Jaccard distance works well for medical data, because having the same symptoms matters more than both not having a symptom.

Overview

In this exercise I took the correlation and regression techniques from Unit 3 and applied them to real macroeconomic data. I used World Bank datasets for global GDP and population from 2001 to 2020 and tried to see whether a country's population size can predict its GDP per capita.

Task A: Correlation

After pre-processing both datasets, which involved handling missing values, calculating means across 2001-2020 and computing per capita GDP, I made scatter plots and looked at the Pearson Correlation Coefficient. The result was a very weak correlation which basically means population size alone tells you almost nothing about how wealthy a country's citizens are. Tiny countries like Luxembourg and huge ones like India are at totally opposite ends of the GDP per capita spectrum regardless of their population.

Task B: Regression

The linear regression model confirmed what the correlation already suggested. The R2 score was very low meaning the model could not really explain the variance in GDP per capita. The residuals also showed a non-random pattern which confirmed that a linear model just does not fit this relationship.

Key Takeaway

This was a really good lesson in feature selection. It showed me that if you fit a model to variables that are barely correlated you get misleading results no matter how fancy the algorithm is. Domain knowledge matters because GDP per capita is driven by things like industrialisation, education and trade policy, not by how many people live in a country. It made me think about the importance of choosing meaningful features rather than just throwing data at a model and hoping for the best.

Link to the GitHub repository: https://github.com/lenasolarova/ML_module

Overview

This unit was about the math behind correlation and regression which are core techniques in ML and also the building blocks for more complex models. I completed four exercises that built on each other progressively.

Exercises Completed

1. Covariance & Pearson Correlation — I explored how two variables move together (covariance) and how to normalise that into a range from -1 to 1 using Pearson correlation. I played around with changing data points to see how outliers and sample size affect the coefficient which was quite interesting to see in practice.

2. Linear Regression — Here I fitted a straight line to data using least squares. I tried adding, removing and shifting data points to see how the line changes, which helped me understand that linear regression finds the line that minimises the sum of squared residuals.

3. Multiple Linear Regression — This extended linear regression to use multiple input variables. It showed how several features can predict an outcome together and also introduced multicollinearity, which is when the input features are correlated with each other and it becomes harder to tell what effect each one has individually.

4. Polynomial Regression — I fitted curved models by adding polynomial terms like x squared, x cubed and so on. Trying different degrees let me see underfitting (too simple) versus overfitting (too complex) which is a trade-off that comes up everywhere in machine learning.

Key Takeaways

Going from simple correlation to polynomial regression helped me understand that the model complexity needs to match the pattern in the data. A linear model underfits curved data and a high-degree polynomial overfits noise. This bias-variance trade-off kept coming up throughout the module, including in the final project where we compared a simpler SVM against a more complex CNN.

I also realised how easily statistical relationships can be misinterpreted. Correlation does not mean causation and an overfit model can make you feel confident in predictions that are actually not reliable. Both of these things have real consequences when you deploy ML models in areas like healthcare or finance.

Link to the GitHub repository: https://github.com/lenasolarova/ML_module

Goal

The goal was to do exploratory data analysis on the Auto-MPG dataset from the UCI Machine Learning Repository. It has fuel consumption data for 398 cars from the 1970s and 1980s with features like engine displacement, horsepower, weight and where the car was made.

Tasks Completed

1. Identify missing values — Only the horsepower column had missing values, just 6 of them (1.5% of the data). They were originally marked as '?' in the raw dataset. I filled them using the median since it handles outliers better.

2. Estimate Skewness and Kurtosis — I calculated these for all numeric features. mpg, displacement and weight showed right-skewness meaning there is a tail of higher values. Most features had pretty normal kurtosis.

3. Correlation Heat Map — This showed strong negative correlations between mpg and weight (-0.83), displacement (-0.80) and horsepower (-0.78). So heavier, more powerful cars use more fuel which is not surprising but nice to see confirmed in the data.

4. Scatter plots for different parameters — I plotted MPG against weight, displacement, horsepower, acceleration, cylinders and model year, all colour-coded by region of origin. You could clearly see that American cars tend to be heavier with lower MPG while Japanese cars cluster at the high-MPG low-weight end.

5. Replace categorical values with numerical values — I mapped the origin column from numeric codes to region names (America=1, Europe=2, Japan=3). The analysis showed Japanese cars had the highest average MPG, then European, then American.

Key Takeaways

This exercise showed me why EDA is such an important first step in any ML pipeline. It helps you spot data quality issues like missing values, understand the distributions and see how features relate to each other. All of which directly affects what model you pick and how you preprocess the data. Even with a small and fairly clean dataset like Auto-MPG you can get some really interesting insights when you explore it properly.

Link to the GitHub repository: https://github.com/lenasolarova/ML_module

Impact of Industry 4.0 on Healthcare Service Availability in Ireland

The transition from Industry 4.0 to Industry 5.0 has significantly transformed the healthcare sector through the integration of technologies such as artificial intelligence (AI), the Internet of Things (IoT), and big data analytics. According to Metcalf (2024), Industry 4.0 focuses on automation, efficiency, and interconnected digital systems, while Industry 5.0 emphasizes a more human-centric, resilient, and sustainable approach. Although these technologies improve operational performance, they also increase dependency on digital infrastructures, making organizations more vulnerable to system failures and cyberattacks.

A significant example is the 2021 ransomware attack on Ireland's Health Service Executive (HSE). The cyberattack caused widespread disruption to healthcare services after critical IT systems were shut down (HSE, 2021). Hospitals lost access to patient records, appointments were cancelled, and diagnostic services were delayed, directly affecting patient care and safety. In addition to operational disruption, the attack created substantial financial costs associated with system recovery and cybersecurity improvements, alongside reputational damage and reduced public trust. Pattnaik et al. (2023) argue that ransomware attacks create wider social and human harms beyond financial losses, particularly when essential public services are disrupted. This was evident in the HSE attack, where patients experienced delays in treatment and reduced access to healthcare services.

This incident demonstrates the importance of Industry 5.0 principles, particularly resilience, cybersecurity, and human oversight. Organizations must ensure that digital systems include strong security measures, backup systems, and manual alternatives to maintain essential services during failures. While Industry 4.0 technologies improve efficiency, Industry 5.0 highlights the need to balance technological advancement with reliability, security, and human well-being.

References

Health Service Executive (HSE) (2021) Cyber-attack and HSE response. Available at: https://www2.hse.ie/services/cyber-attack/what-happened/ (Accessed: 10 May 2026).

Metcalf, L. (2024) Industry 4.0 and Industry 5.0: Implications for modern organisations.

Mikalef, P. et al. (2019) 'Big data analytics and firm performance', Journal of Business Research, 98, pp. 261–276. Available at: https://doi.org/10.1016/j.jbusres.2019.01.044

Pattnaik, N. et al. (2023) 'It's More Than Just Money: The Real-World Harms from Ransomware Attacks', in Furnell, S. and Clarke, N. (eds.) Human Aspects of Information Security and Assurance. IFIP Advances in Information and Communication Technology, vol. 674. Cham: Springer. Available at: https://doi.org/10.1007/978-3-031-38530-8_21

Overview

This project compared a classical machine learning approach (SVM with HOG features) against a deep learning approach (ResNet18 with transfer learning) for detecting pneumonia in paediatric chest X-ray images. The project covered the full pipeline: data preprocessing, model training, evaluation, explainability (Grad-CAM and LIME), bias analysis, ethical considerations, and deployment planning.

Dataset

Chest X-Ray Images (Pneumonia) from Kaggle, originally collected by Kermany et al. (2018). 5,856 images from paediatric patients (ages 1–5) at Guangzhou Women and Children's Medical Centre. Binary classification: NORMAL vs PNEUMONIA with a 2.89:1 class imbalance.

Key Results

Recommendation

ResNet18 was recommended for potential deployment due to higher overall accuracy, fewer false positives, and Grad-CAM explainability that provides visual evidence clinicians can verify. Both models achieved near-perfect recall (>99%), which is critical in medical diagnosis where missing a pneumonia case is far more dangerous than a false alarm.

Skills Learned

Transfer learning, HOG feature extraction, SVM with GridSearchCV, cross-validation, Grad-CAM and LIME explainability, bias analysis, ethical AI considerations, and cloud deployment planning (AWS SageMaker).

Link to the GitHub repository: https://github.com/lenasolarova/ML_module

Leaning oucomes

✅ 1. Identify and manage challenges, security issues and risks, limitations, and opportunities in data wrangling.


✅ 2. Critically analyse data wrangling problems and determine appropriate methodologies, tools, and techniques (involving preparing, cleaning, exploring, creating, optimising and evaluating big data) to solve them.


✅ 3. Design, develop and evaluate solutions for processing datasets and solving complex problems in various environments using relevant programming paradigms.


✅ 4. Systematically develop and implement the skills required to be effective member of a development team in a virtual professional environment, adopting real life perspectives on team roles and organisation.

How were the aims fulfilled

In this module, we shall:

✨ 1. Introduce and review various concepts of big data, technologies, and data management to enable you identify and manage challenges associated with security risks and limitations.

These topics which up to this point have been very abstract to me were reinforced by working on assignments such as the API security review where I had to consider not only the ease of use of an API but also dig deeper into the implications of using an API in a real world project. For me, I had reviewed two APIs I have used in my project at work and evaluating them from the security perspective was a welcomed change of angle as we rarely get to do that.

✨ 2. Critically analyse data wrangling problems and determine appropriate methodologies and tools in problem solving.

This has been an ongoing topic throughout the whole module as data wrangling has been a part of most exercises and assignments. That has made me aware of many different angles and approaches I was unfamiliar with and which I can now apply or even better understand the decision to use them at work which is reinforcing the idea of applying the theory in practice.

✨ 3. Explore different data types and formats. Evaluate various data storage formats ranging from structured, quasi structured, semi structured, and unstructured formats. We explore the various memory and storage requirements.

It has been very interesting to dive deeper into the theory of data types, formats or memory requirements. This has been best highlighted in the team project followed by the individual database build as much of the base laid in the team project when it comes to evaluating proper data types or the right database system for the client, had been used in the final project and thus tested inside a real build.

✨ 4. Critically examine various data collection methods and sources. Review fact finding methods to determine the integrity, reliability and readiness of data extracted and presented for pre-processing, cleaning, and usage.

The data science web scraping activity stands out the most to me as I have previously used regex to scrape data off websites and using a library specifically designed for the task has made the assignment way easier. Yet it was not without its challenges as instead of matching patterns as I would do when using regex, now I had to dive deeper in the html code and find how every particular website used the html tags.

✨ 5. Examine data exploration methods and analyse data for presentation in an organisation. Critically evaluate data readability, readiness, and longevity within the data Pipeline. Examine cloud services, API (Application Programming Interfaces) and how this enables data interoperability and connectivity.

This has been woven into the whole course as in many assignments we had to combine skills regarding data collection and presentation which made it abundantly clear just how easily insights or data can be lost unless proper care is executed. It has been best visible in the final project where all skills we learned thus far were tested and failure or improper handling of data in one stage had catastrophic results.

✨ 6. Examine and analyse the ideas and theoretical concepts underlying DBMS (Database Management Systems) Database Design and Modelling.

The theoretical concepts behind DMBS such as normalisation, which were explored in variety of exercises including the project or a specific exercise on normalisation, were vital for undertstanding how even small chnages in how we store the data may affect the querying. It has also been a good base for undertstanding the trade offs between normalisation and performance or complexity of the design.

✨ 7. Explore the future of use of data and deciphering by examining some fundamental ideas and concepts of machine learning and how these concepts are applied in various methods in handling big data.

Data science is the core of machine learning and artificial intelligence field as a whole so while it was not the main focus of the course, it is crucial to understand how a proper pipeline build can make or break the system consuming the data such as an AI model.