Data Import System

Import your historical farm data from spreadsheets into the Poultry Farm Management System with confidence. Our intelligent import system validates, resolves, and reconciles your data automatically, ensuring data integrity throughout the process.

Overview

The Data Import System transforms CSV files into validated, reconciled database records with full audit trails. It supports seven different types of farm data and includes:

• Automatic validation before any data is committed
• Smart identity resolution to match records with existing entities
• Conflict detection for duplicate or ambiguous data
• Staged commits - review before finalizing
• Complete reconciliation to verify data integrity
• Flock-specific allocation for accurate cost tracking

Supported Data Types

Quick Start Guide

Step 1: Download the Template

Navigate to Import → Templates in your dashboard and download the template for your data type.

Step 2: Prepare Your Data

Open the template in Excel or Google Sheets and fill in your data:

flock_code,breed,acquisition_date,initial_bird_count,source,notes
FL-001,Lohmann Brown,2024-01-15,500,Kenchic Hatchery,Main layer flock
FL-002,Isa Brown,2024-02-20,750,Local Breeder,Second batch

Critical Requirements:

• Keep all column headers exactly as shown
• Use YYYY-MM-DD format for dates (e.g., 2024-01-15)
• Don't add extra columns
• Save as CSV (not Excel format)

Step 3: Upload and Validate

1. Go to Import → Upload
2. Select your CSV file
3. Choose the dataset type
4. Click Upload

The system will:

• ✅ Parse your CSV file
• ✅ Validate every record
• ✅ Show you a preview with any errors
• ✅ Allow you to fix issues before committing

Step 4: Review and Commit

Review the import preview:

• Green rows: Valid records ready to import
• Yellow rows: Warnings (data will import but check carefully)
• Red rows: Errors that must be fixed

Once everything looks good, click Commit Import to finalize.

Import Requirements

File Format Standards

Format: CSV (Comma-Separated Values)

• Encoding: UTF-8 (recommended) or ASCII
• Delimiter: Comma (,)
• Line Endings: Unix (LF) or Windows (CRLF)
• Header Row: Required - first row must contain column names

Date Format: ISO 8601 standard

YYYY-MM-DD

✅ Correct: 2024-06-15
❌ Wrong: 15/06/2024, June 15, 2024, 06-15-2024

Number Format: Use plain numbers without formatting

12500    ✅ Correct
12,500   ❌ Wrong (remove commas)
$12,500  ❌ Wrong (remove currency symbols)

Required vs Optional Fields

Fields marked with * in templates are required. Optional fields can be:

• Left empty
• Omitted entirely (but keep the column header if present in template)

Dataset-Specific Guides

1. Flock Data Import

Import information about your bird flocks or batches.

Required Fields:

• flock_code - Unique identifier (e.g., FL-001, BATCH-A)
• breed - Bird breed name (e.g., Lohmann Brown, Kenbro)
• acquisition_date - When you got the birds (e.g., 2024-01-15)
• initial_bird_count - Starting number of birds (e.g., 500)

Optional Fields:

• source - Where birds came from (e.g., Kenchic Hatchery)
• depopulation_date - When flock ended (leave empty for active flocks)
• depopulation_reason - Why flock ended (e.g., End of cycle)
• notes - Additional information

Example:

flock_code,breed,acquisition_date,initial_bird_count,source,depopulation_date,depopulation_reason,notes
FL-001,Lohmann Brown,2024-01-15,500,Kenchic Hatchery,,,Main layer flock
FL-002,Isa Brown,2024-02-20,750,Local Breeder,,,Second layer batch
FL-003,Rhode Island Red,2024-03-10,300,Farm Hatched,,,Heritage breed
FL-004,Kenbro,2024-04-01,1000,Kenchic Ltd,,,Dual-purpose flock
FL-005,Kuroiler,2023-06-15,450,Local Supplier,2024-06-15,End of cycle,Completed

Best Practices:

• Use consistent flock code patterns (e.g., FL-001, FL-002)
• Import flocks before production or health records that reference them
• Include source information for traceability
• Add notes for context (breeding plans, special characteristics)

Recommended Batch Size: 50-200 flocks per import

2. Production Records Import

Import daily egg collection, feed consumption, and farm metrics.

Required Fields:

• date - Record date (e.g., 2024-06-01)
• flock_code - Which flock (must match existing flock)

Optional Fields (but recommended):

• eggs_collected - Total eggs that day
• eggsLost - Broken/damaged eggs (CSV field name)
• feed_consumed_kg - Feed used in kilograms
• water_consumed_liters - Water consumption
• mortality - Bird deaths that day
• notes - Additional observations

Example:

date,flock_code,eggs_collected,eggsLost,feed_consumed_kg,water_consumed_liters,mortality,notes
2024-06-01,FL-001,420,8,125.5,200,2,Normal day
2024-06-02,FL-001,415,12,124.0,195,0,Hot weather
2024-06-03,FL-001,430,5,126.0,205,1,Production improved

Validation Rules:

• eggsLost cannot exceed eggs_collected
• All numeric fields must be ≥ 0
• Date cannot be in the future
• Flock must exist in system

Best Practices:

• Include all available metrics for better analytics
• Record broken eggs separately for accurate inventory
• Note weather or events that affect production
• Import in chronological order for easier verification

Recommended Batch Size: 500-1,000 records per import

3. Health Records Import

Import vaccinations, treatments, and health events.

Required Fields:

• date - When the event occurred
• flock_code - Which flock was affected
• activityType - Type of health event (see table below)

Activity Types:

• VACCINATION - Preventive vaccines
• MEDICATION - Treatments and medications
• INSPECTION - Regular health checks
• MORTALITY - Death events (auto-updates bird count)
• CULLING - Deliberate removal (auto-updates bird count)
• BIOSECURITY - Biosecurity measures
• OTHER - Other health activities

Optional Fields:

• disease - Disease or condition being treated
• medication - Treatment given
• dosage - Amount administered
• mortalityCount - Number of affected birds (required for MORTALITY/CULLING)
• veterinarian - Vet who performed the service
• cost - Treatment cost
• notes - Additional details

Example:

date,flock_code,activityType,disease,medication,dosage,mortalityCount,veterinarian,cost,notes
2024-06-01,FL-001,VACCINATION,,Newcastle Disease Vaccine,0.5ml per bird,0,Dr. Kamau,15000,Annual vaccination
2024-06-15,FL-001,MEDICATION,Coccidiosis,Amprolium,1g per liter,0,Dr. Kamau,5000,7-day treatment
2024-06-20,FL-002,MORTALITY,Unknown,,,3,,0,Found dead in morning
2024-07-01,FL-001,INSPECTION,,,,0,Dr. Kamau,2000,Routine check-up

Best Practices:

• Always record mortality events for accurate flock tracking
• Include veterinarian names for follow-up
• Document disease names for pattern analysis
• Track treatment costs for financial analysis
• Add detailed notes for future reference

Recommended Batch Size: 200-500 records per import

4. Financial Records Import

Import income and expenses with optional flock-specific allocation.

Required Fields:

• date - Transaction date
• transaction_type - INCOME or EXPENSE
• amount - Transaction amount (positive number)
• description - What the transaction was for
• category - Expense or income category

Optional Fields:

• flockCode - NEW: Specify which flock(s) to allocate to
• payment_method - How payment was made
• reference - Invoice or receipt number
• notes - Additional details

Flock-Specific Allocation 🆕

The flockCode column enables precise cost tracking:

Syntax Options:

1. Single Flock: FL-001

   2025-03-01,EXPENSE,15000,Day-old chicks for A001,FL-001

   Allocates 100% to FL-001

2. Multiple Flocks: FL-001|FL-002|FL-003 (pipe-separated)

   2025-03-15,EXPENSE,20000,Feed for three flocks,FL-001|FL-002|FL-003

   Splits cost proportionally across the three flocks

3. Farm-Wide (empty or omit column)

   2025-04-01,EXPENSE,5000,Electricity bill,,

   Allocates to all active flocks automatically

Complete Example:

date,transaction_type,amount,description,category,flockCode,payment_method,reference,notes
2025-01-15,EXPENSE,75000,Day-old chicks - Layer A001,FLOCK_STOCKING,FL-A001,M-PESA,INV-1001,500 Lohmann Brown chicks
2025-02-01,EXPENSE,45000,Feed delivery,FEED,FL-A001|FL-C003,BANK_TRANSFER,PO-2024,Split between two flocks
2025-02-15,INCOME,28000,Egg sales to Njeri,EGG_SALES,FL-A001,CASH,SALE-301,60 trays @ KES 350
2025-03-01,EXPENSE,8000,Electricity bill,UTILITIES,,M-PESA,KPLC-4567,Farm-wide expense

Flock Code Validation:

• ✅ Valid code → Allocates to specified flock(s)
• ⚠️ Invalid code → Falls back to farm-wide allocation (logs warning)
• ⚠️ Mixed valid/invalid → Uses only valid codes
• ℹ️ Empty → Farm-wide allocation (all active flocks)

Category Options:

Expense Categories:

• FEED - Bird feed purchases
• MEDICATION - Medical supplies and treatments
• UTILITIES - Electricity, water, gas
• LABOR - Wages and salaries
• EQUIPMENT - Tools and machinery
• FLOCK_STOCKING - Purchasing birds (day-old chicks, pullets)
• MAINTENANCE - Repairs and upkeep
• VETERINARY - Vet services
• TRANSPORT - Vehicle and delivery costs
• OTHER_EXPENSE - Miscellaneous expenses

Income Categories:

• EGG_SALES - Selling eggs
• BIRD_SALES - Selling birds (live or processed)
• MANURE_SALES - Selling fertilizer
• OTHER_INCOME - Miscellaneous income

Best Practices:

• Use consistent category names for better reporting
• Always include flock codes for production expenses (feed, chicks, medication)
• Leave flock code empty for farm-wide costs (electricity, rent, office supplies)
• Track payment methods for cash flow analysis
• Include reference numbers for audit trails

Recommended Batch Size: 200-500 transactions per import

5. Customer & Supplier Data

Import contact information for buyers and vendors.

Customer Fields:

• name (required) - Customer name
• phone - Contact phone
• email - Email address
• address - Physical address
• notes - Additional information

Supplier Fields:

• name (required) - Supplier name
• phone - Contact phone
• email - Email address
• tax_id - Tax identification number
• address - Physical address
• supplier_type - Type of supplier (e.g., FEED, CHICKS, EQUIPMENT)
• notes - Additional information

Example - Customers:

name,phone,email,address,notes
Njeri's Shop,0722123456,njeri@email.com,Nairobi CBD,Regular egg buyer - 20 trays weekly
Kamau Wholesalers,0733456789,,Thika,Large orders monthly
Fresh Eggs Ltd,0744789012,orders@fresheggs.co.ke,Westlands,Premium buyer - pays on time

Example - Suppliers:

name,phone,email,tax_id,address,supplier_type,notes
Kenchic Limited,0700123456,sales@kenchic.co.ke,P051234567M,Nairobi,CHICKS,Day-old chicks supplier
Pembe Mills,0711234567,info@pembemills.com,P051234568N,Kitale,FEED,Feed supplier - 10% discount
AgriVet Store,0722345678,,P051234569O,Local,MEDICATION,Local veterinary supplier

Best Practices:

• Include phone numbers for quick contact
• Add email for digital invoices
• Note payment terms in notes field
• Tag VIP customers for special treatment
• Categorize suppliers by what they provide

Recommended Batch Size: 100-300 contacts per import

6. Weight Records Import

Track bird growth over time.

Required Fields:

• date - When birds were weighed
• flock_code - Which flock was sampled
• average_weight - Average weight in grams or kg

Optional Fields:

• sample_size - How many birds were weighed
• notes - Observations

Example:

date,flock_code,average_weight,sample_size,notes
2024-06-01,FL-001,1850,50,Week 24 - target weight reached
2024-06-08,FL-001,1920,50,Week 25 - steady growth
2024-06-15,FL-001,1975,50,Week 26 - good condition

Best Practices:

• Weigh same sample size consistently
• Record same day of week for comparisons
• Note any unusual observations
• Track multiple flocks separately

Recommended Batch Size: 200-500 weight records per import

Import Process Deep Dive

The Six-Stage Pipeline

Our import system processes data through six carefully designed stages:

CSV File → Parse → Validate → Resolve → Allocate → Commit → Reconcile

Stage 1: Parse

• Reads CSV file
• Detects encoding and delimiter
• Extracts all rows
• Identifies column headers
• Creates staged records

Stage 2: Validate

• Checks required fields present
• Validates date formats
• Verifies number ranges
• Applies business rules
• Flags errors and warnings

Stage 3: Resolve (Identity Resolution)

• Matches flock codes to existing flocks
• Finds or creates customers/suppliers
• Links transactions to correct entities
• Detects ambiguous matches
• Creates resolution conflicts for review

Stage 4: Allocate

• For financial records only
• Determines allocation type (manual vs auto)
• Calculates proportional distributions
• Applies flock-specific allocations
• Logs allocation decisions

Stage 5: Commit

• Inserts records into database
• Updates bird counts (for mortality)
• Creates allocation entries
• Maintains referential integrity
• Executes in transaction (all-or-nothing)

Stage 6: Reconcile

• Verifies record counts match
• Checks date ranges
• Validates relationships
• Compares totals
• Generates reconciliation report

Understanding Import States

Batch States

Record States

Error Handling & Troubleshooting

Common Validation Errors

Error: "Required field missing: flock_code"

• Cause: Required column is empty
• Fix: Fill in the missing value or remove the row

Error: "Invalid date format: 15/06/2024"

• Cause: Date not in YYYY-MM-DD format
• Fix: Change to 2024-06-15

Error: "Flock not found: FL-999"

• Cause: Referenced flock doesn't exist
• Fix: Import the flock first, or correct the flock code

Error: "eggsLost (12) cannot exceed eggs_collected (10)"

• Cause: Broken eggs more than total collected
• Fix: Check your numbers - likely a typo

Error: "Negative value not allowed: amount"

• Cause: Negative number in amount field
• Fix: Use positive numbers only (transaction type indicates income vs expense)

Resolution Conflicts

Conflict: Multiple customers match "John"

Found 3 matches:
1. John Kamau (0722123456)
2. John Mwangi (0733456789)
3. John Njoroge (0744789012)

How to Resolve:

1. Review the conflict in Import Dashboard
2. Select the correct match manually
3. Or add unique identifier to CSV (phone number)
4. Re-upload with more specific information

Import Failed? Here's What to Do

1. Check the error log

   • Go to Import → History
   • Find your batch
   • Click "View Details"
   • Read the error messages

2. Common causes:

   • File encoding issue (save as UTF-8)
   • Extra columns added
   • Commas in values (use quotes: "value, with comma")
   • Date format incorrect

3. Fix and retry:

   • Download the failed records
   • Fix the issues
   • Upload again (same batch)
   • The system will only import new records

Best Practices & Tips

Before You Import

✅ Clean your data first

• Remove extra columns
• Fix typos and formatting
• Standardize naming (e.g., all flock codes use same pattern)
• Check dates are realistic

✅ Start small

• Test with 10-20 records first
• Verify they import correctly
• Then import the rest

✅ Import in order

1. Flocks first
2. Then production, health, weight records
3. Customers and suppliers
4. Finally financial transactions

During Import

✅ Review carefully

• Check the preview before committing
• Verify dates and amounts
• Confirm flock allocations look right
• Read all warnings

✅ Don't rush

• Take time to understand errors
• Fix issues properly
• Better to get it right than fast

After Import

✅ Verify your data

• Check a few records in the system
• Compare totals with your spreadsheet
• Run a report to spot any issues

✅ Keep your CSV files

• Archive imported files
• Include import date in filename
• Useful for future reference

Performance Tips

Optimal File Sizes:

• Flocks: 50-200 records
• Production: 500-1,000 records
• Health: 200-500 records
• Financial: 200-500 transactions
• Customers/Suppliers: 100-300 contacts

Why smaller batches?

• Faster validation
• Easier to review
• Quicker to fix errors
• Less memory usage

FAQ

Q: Can I edit records after importing them?
A: Yes! Once imported, records become regular system data. You can edit or delete them through the normal UI.

Q: What happens if I import the same data twice?
A: The system detects duplicates using unique identifiers (flock codes, dates, etc.) and won't create duplicate records.

Q: Can I import data for multiple farms at once?
A: No, imports are organization-specific. You need to import data separately for each farm/organization.

Q: How do I handle missing flock codes in old records?
A: Leave the flockCode field empty - the system will allocate to all active flocks automatically.

Q: Can I undo an import?
A: There's no automatic undo, but you can manually delete imported records. Best practice: review carefully before committing.

Q: What's the maximum file size?
A: We recommend files under 5MB (typically 5,000-10,000 records). Larger files should be split into batches.

Q: How long does import take?
A: Parsing and validation are nearly instant. Commit time depends on record count - typically 1,000 records commit in under 5 seconds.

Q: Can I schedule automatic imports?
A: Not currently. All imports require manual review and approval for data quality.

Support & Resources

Need Help?

• Documentation: Complete guides at /docs/features/data-import
• Templates: Download at /import/templates
• Import History: View past imports at /import/history
• Contact Support: support@poultryfarmsaas.com

Related Guides

• Flock Management - Managing flocks
• Financial Tracking - Financial reports
• Cost Allocation - Cost allocation workflows
• Lifecycle Templates - Choosing the right production system

Conclusion

The Data Import System transforms spreadsheet data into structured farm records with minimal effort. By following this guide and using our templates, you can confidently migrate years of historical data while maintaining complete data integrity.

Remember:

• ✅ Start with small test imports
• ✅ Import flocks before dependent records
• ✅ Review carefully before committing
• ✅ Keep your CSV files for reference
• ✅ Verify data after import

Happy importing! 🐔📊