FLEX Platform Data Guide
Introduction
Welcome to the Recurve FLEX Platform Data Guide! This guide contains helpful information on the Recurve FLEX Platform data integration process and is intended as a resource for program administrators and technical teams alike. This document aims to clarify data requirements and expectations to promote seamless data setup and ongoing data upload process, which will enable accurate program insights.
Data Integration Overview
To successfully integrate data into the Recurve FLEX Platform, data will need to meet the data requirements explained in this guide. See below for a high-level overview of the integration and ongoing refresh process.
-
Data Requirements
- All data intended for the Recurve FLEX Platform must follow strict standards/conventions in order to be fit for ingestion.
- These standards are outlined in the Client Data Schema and include specific requirements related to which fields must be included and what values are acceptable.
- Once data is conformed to the required standards, data files must follow a strict naming convention to drive automated process (e.g., SFTP automation).
-
Data Transmission
- The client uploads data to Recurve’s FLEX Platform SFTP.
-
Data Collection and Processing
- Once data has been uploaded to the Recurve FLEX Platform, the data is processed and integrated into the FLEX Platform.
-
FLEX Platform Launch and Ongoing Refreshes
- Users will be able to log in and access the latest processed data available on FLEX Platform.
- Steps 1-3 will look the same for initial platform launches and ongoing refreshes.
Data Requirements
This section identifies data and formatting requirements. These requirements are important for ensuring data integrity, accuracy, and consistency for every data transmission. The FLEX Platform data requirements are outlined below.
There is a minimum requirement of data that every customer must upload to be able to launch the FLEX Platform, for example, Meter and Site data are required for every product feature offered. In addition to the base required data, feature-based data requirements like Project and Event data can be provided to gain additional insights.
The following chart provides an overview of the categories of data you will need to provide based on the features you would like to enable.
| Feature | Data Requirements | ||||
|---|---|---|---|---|---|
| Meter Data | Site Data | Project Data | Event Data | How much data do I need? | |
| Targeting | X | X | 1 year | ||
| Measurement: Tracking Backcast Evaluation Marketplace (Pay for Performance) | X | X | X Note: Alternatively project data can be collected and enrolled with the FLEX platform. | 1 year prior to the intervention, ongoing after the intervention, but may vary by program requirements. | |
| Demand Response: Backcast Evaluation Marketplace | X | X | X | 45 days prior to the event, 15 days after the event, but this may vary by program requirements. |
Note: Meter, Site, Project, and Event data are described in more detail below:
Recurve Client Data Schema
Data transmissions must match the Recurve Client Data Schema format for the Recurve FLEX Platform to test, accept, and process the data.
The Recurve Client Data Schema provides granular, column-level details of all of the data fields that you could provide to Recurve.
Data Transmission
The Recurve FLEX Platform data transmission method and required naming conventions promote clarity and maintainability. The following data sharing and storage structures are essential in making sure that data can be managed efficiently and effectively.
Transmission Method
The Recurve FLEX Platform uses Recurve’s Secure File Transfer Protocol (SFTP). All clients will need to provide an SSH key for Recurve to furnish them with an account. If you need assistance with this process, please contact our support team.
The Recurve SFTP Connection Guide can be found here.
Transmission File Naming Convention & Format
Data files should be named based on the contents of the file(s). Consistent file naming improves communication and ensures that everyone involved at this stage of data intake can easily identify and refer to specific files.
- File Naming Convention Best Practices: [file_type_identifier]__[timestamp_with_underscores].csv
- Example: service_point_reads__2024_11_12_13_01_01.csv
Files must be provided in CSV format.
Data Resolution Defaults
Based on Recurve’s experience processing Meter/AMI data, there are several categories of common issues that can appear in this category of data. Recurve’s FLEX Platform has a default methodology for resolving these issues.
Recurve’s FLEX Platform will scan your data for these problem categories and resolve them using the default methods described below. Then, it will scan the data again to see if new categories have emerged.
| Problem Type | Default Resolution |
|---|---|
| Duplicate reads | Removed |
| Conflicting reads | Take smallest non-zero absolute value read |
| Intervals larger than daily (i.e. billing/monthly) | Upsample to daily |
| Intervals smaller than hourly (i.e. 15 min) | Downsample to hourly |
| Meter data with mixed intervals | Downsample to largest interval |
| Baseline insufficient | Flag as likely to be disqualified |
If you need an alternative resolution methodology, Recurve’s intake process is highly configurable. Custom intake resolution rules can be scoped with your account representative.
Repercussions of Inconsistent Data
It is understood that formats for certain fields may vary, however, consistently meeting the data requirements is necessary to ensure rapid results and insights into your data.
Failing to provide consistent data can look like the following:
-
- Changes to the data format after agreeing upon the schema and therefore do not meet agreed-upon
-
- Delays in data transfer
-
- The data transfer is missing data
If you are not able to meet these requirements, please contact your Sales Representative or Strategic Account Manager for assistance.
The Two Primary Types of Data
The Recurve Data Intake process has two broad categories of data:
-
Meter Data: Meter data is time series data, often referred to as “AMI data” when smart meter data is available. Meter level data is required for any/all measurements and describes the usage of a certain type (electricity or gas) over time at a specific point.
-
Curated Attributes: Curated Attributes are other types of data that describe Service Points in some way. Curated Attributes are broken up into several categories/groups but always refer back to a Meter at a row level. Examples of Curated Attributes:
- Site Data: Describes where a meter is located. These are required for any/all measurements. Some manner of locational data is required for mapping weather data to the Service Point.
- Customer Data: Describes information about the customer consuming/producing energy at the meter. An example would be a sector (i.e. Commercial, Residential, Industrial, or Agricultural) or perhaps a NAICS code. These are not always required to perform measurements but can be useful when you want to group or filter measurements in various ways.
- Project Data: Required if you are measuring savings. These describe things that are expected to change consumption at a meter, such as intervention/measure details and blackout dates.
- Event Data: Required if you are measuring event-driven savings. These describe details about the events called, such as start and end timestamps of the events. In addition, enrollment data about which customers were being dispatched would be required.
Not all fields are always required when it comes to Curated Attributes and some might only be useful if there is a valuable insight you are trying to gain as part of the measurements. For example - if you are measuring savings and trying to find what category of intervention (HVAC only, Lighting only, etc) is proving most effective, you may want to include the category field allowed by the Curated Intervention Attribute, even though this field is not required to perform the measurements themselves. See the sections below and the Recurve Intake Schema for detailed column-level specifications.
Three Primary Uses for Curated Attributes
While there are 2 primary types of data at a high level - Meter and Curated Attributes - there are 3 primary uses for Curated Attributes themselves.
-
Grouping measurement results - A Curated Attribute can be used to aggregate results. An example of this would be using a Curated Campus Attribute to show a dozen Meters - potentially part of multiple buildings - being part of a more broadly defined “campus” that you are interested in measuring. The calculations would still be performed at the Meter level before being grouped into the Campus.
-
Providing columns for clustering/comparison groups - Curated Attributes can be used to help define comparison group matching. Common columns for this purpose include sector, climate zone, NAICS Group, and solar PV status, which ensures participating Meters are compared against relevant non-participants.
- Learn more about our comparison group methodology here: GRIDmeter™ Methodological Overview
-
Providing filters for additional post-measurement analysis - Curated Attributes can be transformed into useful filters in your Recurve products that you can use to slice and dice results. Want to know which types of measures are performing best? Which contractors? Define this need with your account representative and provide these Curated Attributes in your input data and it can be accommodated.
What to Provide if You Want to Use Filters
Curated Attributes used as filters provide an exciting opportunity to get additional insights. As an example, say you want to see which intervention type is achieving the highest savings.
For our hypothetical use case above, the Project data is already defined with a “category” column that fits this need. You will then need to provide an enumeration of the categories you expect. Do you want your data to be mapped to 3 distinct categories, perhaps “HVAC, Lighting, and Other”? If you can provide the data already in those 3 categories, the lift at Recurve is minimal. However, if your input data contains dozens of categories that need to be mapped to these 3, you will need to help provide the means to perform that mapping. Recurve can work with you to determine the time/cost of such exercises.
Site Data
Site data can be as simple as two columns - a Meter ID and an associated address, which Recurve will then geocode.
Meters must be associated with latitude and longitude so that proper weather data can be associated with the Service Point Reads.
Ideally, latitude and longitude are provided. However, Recurve also uses third-party geocoding services to translate address strings into latitude and longitude coordinates.
In cases where an entire address string can not be provided, at minimum, a zip code can be used. In cases where a zip code is used, the centroid latitude and longitude will be used. In areas with highly variable climate zones (due to mountains or other geographical features), this is not optimal as the zip code centroid may fall outside of the correct climate zone for the meter. For this reason, the more granular the location, the better the measurement.
Project Data
Project data is only required when performing savings calculations. Like site data, the minimal required columns are simple - a meter, a date the work on the intervention started, and a date the work was completed. If using Project Hub for enrollment, project data can be collected and used directly from the Recurve App database.
To determine the impact of a single intervention at a meter where there have been multiple interventions, there must be 12 months between the interventions. This means if multiple measures occurred at a meter, those measures must ultimately be rolled up to a single intervention. The impacts of such a rolled-up intervention can not be easily disaggregated.
Event Data
Event data is only required when performing demand response calculations. Like project data, the minimal required columns are simple - a meter, a start time for the event, and an end time for the event.
You may find it easier to provide a list of the events that occurred and separately a list of the Meters that participated in those events. This is perfectly valid and Recurve will handle the mapping in that case.
Other attributes you can provide as part of the event include fields like the program name, device type, and others.
Other Curated Attributes
Consult the full Recurve Intake Schema for the full list of supported attributes. These can be used for any cases described above in Three Primary Uses for Curated Attributes.
Consult your account representative on how your data can be used to unlock additional insights with Recurve tools.
Security
Recurve is SOC2 compliant and is committed to protecting customer data through internal controls, policies, and procedures.
Recurve’s Security team can furnish you with a copy of our SOC2-compliant security reports and various other security measures Recurve has adopted by request.
Types of reports:
- SOC 2 Type 1 Reports document that Recurve has policies and procedures in place to meet the relevant SOC 2 controls as a company.
- SOC 2 Type 2 Reports demonstrate that Recurve, as a company, has implemented policies and procedures.