Data Feeds via SFTP

Schedule data imports to automate the creation and updates of data from an external data source.

Scheduled Data Imports

LCvista offers the ability to work with your account manager to set up a scheduled data import to automate the creation and updates of data from an external data source. Data sources that can be created/updated include various objects such user accounts, user jurisdictions, programs, sessions and session records.

For example, you may have user account data from your HRIS application that you want to update nightly in to LCvista. This can be accomplished by creating an Excel (.xlsx) or CSV file from your HRIS system in a predetermined format (see File Format), to an SFTP site where LCvista will automatically process the file.

What is the difference between SFTP and FTP?

FTP stands for File Transfer Protocol and SFTP stands for SSH File Transfer Protocol. Both are common ways for users to download or upload files to the Internet. With SFTP, data is securely transferred using encryption and no clear text file data is transferred. If choosing between SFTP or FTP, we strongly recommend that you use SFTP because it’s a more secure way to transfer files.

SFTP Host Configuration

The choice of an SFTP host is usually made as part of implementing a new scheduled data feed. Please work with your account manager of your preference for a host. The following FTP configurations are supported for this type of integration:

LCvista hosted FTP

LCvista partners with a 3rd party to provide secure FTP hosting. If your company does not have an FTP host or you would like to use LCvista's provider please work with your account manager to get an account created.

External hosted FTP/SFTP

LCvista is able to connect to a standard FTP host that accepts SFTP protocol. LCvista currently only supports username/password credentials when accessing an external host.

Key Concepts to Configure a Data Feed

The following items are critical to the success of your data feed. Please review each of these with your account manager to ensure the success of your data feed.

File Names

Note: Each file name must be unique.

There are a few reasons LCvista requires a unique name, however the most important is that it provides ability to trace imports to a specific file. Typical naming conventions follow a pattern that describes the data being updated and includes a timestamp such as YYYYMMDD_HHMMSS in the filename to ensure the file name is unique.

Here are some example of what a unique filename generated daily at noon may look like:

user_accounts_20201115_120000.xlsx
user_accounts_20201116_120000.xlsx
user_accounts_20201117_120000.xlsx
user_accounts_20201118_120000.xlsx

Incremental Load vs Full Load

Depending on the type of data being loaded, you'll need to determine the logic used to extract the file from your external system. It is highly recommended that incremental (or delta) logic is used to extract data in all sources except user account file feeds. A user account data feed that creates and updates accounts in LCvista can be run as a full load.

If you are loading programs, sessions, or session records you'll want to reduce the amount of data reprocessed with each load. The most ideal scenario is to configure your extract to only extract new or updated records that have been created since the last extract ran. If that is not possible you can extend a look back to include some additional hours of data. For example, if you are running a session record extract every 8 hours then it can include 12 hours of data (a 4 hour look back) to ensure no data updates are lost.

Archiving Files

LCvista will manage archiving processed files on LCvista hosted FTP sites.

It is up to the client to move/archive, delete, rename, etc. as necessary after processing. LCvista will not delete, move, or rename files on externally hosted FTP servers. Keep in mind that LCvista does log file names and will not process the same file twice, so files do not need to be cleaned up immediately after each job run. However it is important the client manages files to ensure proper disk space on their FTP server.

Supported Data Feeds

User Accounts

Create and/or Update user accounts based on matching username.

Download a user account sample template

Jurisdiction Person

Create and/or update the jurisdictions associated to a user's account. It is recommended that this data feed is only used to create new jurisdictions on an account. There is an option with this data feed to ignore updates if a jurisdiction already exists on an account.