IDMC data systems development and maintenance

BACKGROUND

The Internal Displacement Monitoring Centre (IDMC) is the world’s definitive source of data and analysis on internal displacement. Since our establishment in 1998 as part of the Norwegian Refugee Council (NRC), we have offered a rigorous, independent and trusted service to the international community. Our work informs policy and operational decisions that improve the lives of the millions of people living in internal displacement, or at risk of becoming displaced in the future.

With a team of almost 40 people in Geneva, we monitor more than 252 countries in the world for which we provide verified, consolidated and multi-sourced estimates of the number of people internally displaced or at risk of becoming displaced by conflict, violence, disasters. Our data and evidence are published on our website and via our Global Internal Displacement Database.

Currently, IDMC’s core systems are essential for maintaining the Global Internal Displacement Database (GIDD) and enabling data-driven insights on internal displacement trends worldwide. To enhance and modernize its data infrastructure, IDMC is rolling out a three-phase transition plan. These terms of reference focus on the first phase of the transition plan implemented by IDMC.

Phase I of IDMC’s Transition Project is focused on preparing IDMC’s data systems and applications for work by multiple service providers. Key objectives include finalizing the migration from legacy systems that need to be decommissioned, into new systems currently in use, maintaining operational continuity, and establishing robust documentation.

CONSULTANCY OBJECTIVE

The primary objective of Phase I is to prepare for a transition of IDMC’s data infrastructure and applications to a multiservice provider model, ensuring the separation of operations and development functions. This approach is designed to foster collaboration and accountability among service providers, reduce the dependency on any single vendor, and ensure that systems remain operational, secure, and scalable.

CONSULTANCY TIME and IMPLEMENTATION TIMEFRAME

This is a consultancy of 4 months

Description of services required:

• Data migration and validation
  • Complete the migration of all data from deprecated systems to new systems without any data loss or downtime during core working hours (CET).
  • Correct and re-migrate specific data elements (e.g., location names, coordinates, dates) for accuracy.
  • Successfully migrate reviews and source files, ensuring data integrity.
• System operations and maintenance
  • Upgrade serverless databases for improved performance and scalability.
  • Update all critical libraries and dependencies to the latest versions to maintain system stability.
  • Implement automated health checks and monitoring (e.g., via Sentry) to ensure system performance and timely issue detection.
• Disaster recovery
  • Develop and perform stress tests for disaster recovery scenarios, confirming recovery processes meet the required response times.
  • Draft a detailed recovery and maintenance runbook with clear, actionable steps for use during system outages.
  • Create a comprehensive Business Continuity Plan to maintain essential operations during major disruptions, covering resource allocation, communication, and incident management.
• Decommissioning legacy systems
  • Decommission outdated systems post data migration (deprecated platforms).
  • Migrate any remaining services from Linode to AWS, ensuring a streamlined and cost-effective infrastructure.
• Support independent deployment capability
  • Full documentation and support for IDMC independent deployment capacity.
  • Establish deployment protocols for minor and major updates, enabling IDMC to perform system updates without external service provider support.
  • Conduct final QA and troubleshooting for IDMC’s independent deployments, ensuring a seamless handover to multiple service providers.
• Comprehensive documentation
  • Develop system documentation in GitHub, including code functionality, deployment, and release notes.
  • Prepare system architecture and configuration management documentation in the knowledge base.
  • Documentation of products, processes, software change deployments and workflows to enable knowledge transfer.
  • Compile user guides and troubleshooting records for administrators and end-users to ensure smooth onboarding and issue resolution.

Preferred technical qualifications and expertise of the vendor:

For the successful implementation of Phase I of IDMC’s transition project, the vendor should demonstrate the following qualifications and expertise:

• Data migration and systems integration
  • Demonstrated expertise in high-volume data migration with complex data structures, including experience in maintaining data integrity and consistency during live migrations.
  • Proficiency with data management systems and deep technical knowledge of PostgreSQL databases, including advanced SQL and stored procedure usage, to ensure efficient and accurate data transfers.
  • Proven experience in API-based data synchronization and ETL processes to support batch data migration, with an emphasis on consistency checks, error handling, and reconciliation.
• Backend development and cloud infrastructure
  • Strong expertise in backend development frameworks, particularly Django (Python) for server-side applications, as well as GraphQL for APIs.
  • Proven experience with AWS infrastructure, including Aurora Serverless, S3, EC2, and CodePipeline, with a focus on high availability, security, and scalability.
  • Experience in optimizing cloud-based systems for cost efficiency, performance, and reliability.
  • Strong skills in Terraform and AWS CloudFormation to automate the provisioning and management of cloud resources, ensuring infrastructure is secure, efficient, and reproducible.
• Frontend development and user interface management
  • Advanced skills in React for developing and maintaining complex, responsive user interfaces that meet IDMC’s specifications.
  • Experience building, managing, and optimizing modular components, integrating with GraphQL APIs, and implementing best practices for performance, accessibility, and security on user-facing applications.
• Monitoring, disaster recovery, and continuity planning
  • Expertise in setting up monitoring solutions like Sentry and AWS CloudWatch with custom alerts and logging configurations for rapid identification of anomalies and issue resolution.
  • Proven ability to develop and test robust disaster recovery plans, including high-fidelity simulations for various failure scenarios and complex recovery workflows.
  • Extensive experience creating runbooks, continuity plans, and automated failover systems to ensure uninterrupted system availability and facilitate seamless incident response.
• Legacy system decommissioning
  • Expertise in identifying and safely decommissioning legacy systems, without impacting active services.
  • Familiarity with strategies for reducing technical debt, including system refactoring, dependency management, and library updates.
• Documentation
  • Experience with industry-standard documentation practices, including code, API, and deployment documentation in GitHub.
  • Ability to create comprehensive system documentation, including configuration management, system architecture, and user/admin guides.
  • Skilled in developing onboarding materials for future service providers, ensuring a smooth and effective handover.
• Security and compliance
  • Advanced experience implementing cloud security best practices, including identity management, network security.
  • Skilled in embedding security assessments, vulnerability scanning, and automated security checks within GitHub Actions and CI/CD pipelines to proactively identify and mitigate potential threats.
• Quality Assurance (QA) and Quality Control (QC) management:
  • Establish a robust Quality Assurance and Quality Control process, ensuring that every stage of the transition, from data migration to system updates, meets industry standards and IDMC’s specific requirements.
  • The company’s QA team should be in-house and dedicated to continuous testing and validation throughout the deployment cycle. Key components include automated testing in GitHub Actions, code reviews, and staging environment validations.
  • Implement a detailed QA workflow to address troubleshooting and remediation quickly, minimizing any risk of downtime or data integrity issues.
• Preferred certifications and experience
  • Certifications: AWS Certified Solutions Architect, Certified Django Developer, or similar credentials in cloud and web development.
  • Sector experience: Prior work with non-profit, humanitarian, or international organizations, especially those dealing with data-intensive projects and global data reporting, is an advantage.
  • Language proficiency: Proficiency in English for seamless communication with IDMC’s global team.

Management

The vendor selected will be managed by a Project Manager within IDMC, that is part of the Data management and risk analysis Hub.

Kindly refer to the Terms of Reference for a full description of the consultancy, including details regarding application process and relevant annexes. Only complete proposals will be considered.