Tasks & Achievements

• managing and monitoring ETLs on cloud using Airlfow and Grafana
• Reviewing error logs on Elastic Search and AWS CloudWatch
• Designed and implemented two data qualities using OOP Python and a hybrid Redshift-MySqL environment
• Dealing with six-digit-size databases on AWS Redshift
• Reading from daily dashboards and reporting from Tableau
• Developing efficient and complex queries to join large tickets’ sales tables
• excelled in using Airflow sensors to orchestrate Kubernetes pods

Tech stack:

• AWS Redshift + MySQL
• Python
• Airflow
• Docker
• Grafana + Tableau
• Elastic Search

To offer data solutions to the Spanish governmental bodies that are opting in the open-data and transparency program in Spain.

Achievements:

• Architected an ETL dataflow on NIFI to enquire Oracle and upload results to CKAN on EC2.
• Helped a Spanish entity revive their website by increasing its EBS size.
• Debugged GCP python functions in visual studio using functions framework integration.
• Found a way to migrate AWS EC2 instances across accounts belonging to different organisations using AMI and Amazon Backup
• Adopted TablePress for creating editable and user-friendly data tables
• Modified user permissions to allow secure access/edit to Postgres micro-service in Ubuntu

Tech stack:

• Docker & Docker-compose
• CKAN
• AWS
• Python
• Apache NIFI
• Postgres & MySQL
• WordPress

Working mainly on Eyepix ML models and CI/CD, and cooperating with the Backend and business analysts to offer reliable solutions to the end-user. Achievements:

• Leading the DevOps/MLOps of Eyepix project (eyepix.net).

  • Migrated all micro-services to use Azure Container Instances, Service Bus, and Redis.
  • Also same services can run locally using RabbitMQ, docker, and local Redis.
  • Developed a single docker-compose to manage 14 mirco-services with a single config file.
  • Implemented a one-line switching mechanism (Azure / on-premises).
  • Demonstrated notable stamina as my team and I tested and deployed 5 DL modules in 1.5 months.
  • Reviewed and tested several modules, i.e. fire detection, asset guardian, and heatmap.

• Contributed to petner.net.

  • Classified dogs and cats into 175 breeds using Azure Custom Vision (Petner application).

Achievements:

• Analysed statistically a clinical validation by two radiologists to evaluate the realism of 150 in-house GAN-synthesised and real mammography lesions.
• Achieved high levels of realism in the synthetic images that two expert radiologists classified 45% of them as real.
• Co-authored two papers in SPIE Medical Imaging conference and IWBI workshop.
• Reviewed Radiomics and Deep Learning for Triple Negative Breast Cancer classification in mammography.
• Co-mentored and trained an MSc thesis intern on Machine Learning for medical image analysis; designed two data engineering assignments to help the intern develop problem-solving techniques.
• Won the €55K Catalan FI 2020 grant.
• Funds: FI 2020 grant, SMARTER project (Ref. DPI2015-68442-R), and ICEBERG project (Ref. RTI2018-096333-B-I00).

Outcome:

• First author, SPIE Medical Imaging conference paper (DCGANs for Realistic Breast Mass Augmentation in X-ray Mammography).
• First author, IWBI workshop paper (Quality analysis of DCGAN-generated mammography lesions).

GAN (Generative Adversarial Networks) for realistic data augmentation and lesion simulation in x-ray breast imaging using Pytorch. The Hologic mammograms were acquired from OMI-DB dataset (OPTIMAM).

Responsibilities include:

• Data filtering and preprocessing.

  • Leaving out outliers.
  • Histogram normalisation.
  • Random patch extraction from breast tissue and lesions (if applicable).

• Deep Convolutional GANs implementation (Pytorch).

  • Training and fine-tuning the hyper-parameters.
  • Validating the models using k-fold cross-validation.
  • Testing using unseen cases (lesion/tissue classification).

Achievements:

• Developed Generative Adversarial Networks for realistic data augmentation in X-ray breast imaging (details).
• Generated 2,000 realistic, diverse mammographic lesions using the trained GANs.
• Boosted imbalanced classification from 87% to 96% F1 score by incorporating synthetic cancer lesions in the dataset.
• Employed t-distributed Stochastic Neighbour Embedding for 2D visualisation of high-dimensional data using Matplotlib.
• Improved data homogeneity through filtering outliers and histogram normalisation using Python.
• Excelled in training and finetuning GANs to avoid divergence and mode collapse using PyTorch.

Supervisors: Dr Robert Marti, Dr Oliver Diaz.

Researching the methods of matching between high- and low-quality dermoscopic images including:

• Feature detection and description (using SIFT)
• Deep Learning (Siamese network and face detection).

Employer: Prof Rafael Garcia (rafael.garcia@udg.edu)