Release notes
Version 3.8
See the 3.8-beta1 release page.
Version 3.7
Vulnerabilities
- (from 3.7.8 on) The fix of the vulnerability below was fixed, as it did not prevent the exploit using a particular URL syntax for the JDBC server. The vulnerability was assigned identifier CVE-2024-23833. We thank @l0n3rs for disclosing it to us.
- (from 3.7.5 on) A moderate vulnerabilities in the database extension was fixed. Connecting to a malicious MySQL server could read files or execute arbitrary code on the machine running OpenRefine. The vulnerabilities were assigned CVE-2023-41886 and CVE-2023-41887 identifiers respectively, and were reported by @nbxiglk0.
- (from 3.7.4 on) A moderate vulnerability in project import was fixed. Importing a maliciously crafted project could execute arbitrary code on the machine running OpenRefine. This vulnerability has been assigned the CVE-2023-37476 identifier. It was reported by Stefan Schiller from SonarSource.
New features
- Most text exposed to users in OpenRefine's UI can now be translated. Some strings (generated server-side) were not translatable so far. To help translators catch up on this backlog, do not hesitate to join us on Weblate. (#5030)
- New media files can be uploaded to Wikibase instances such as Wikimedia Commons. The wikitext of existing files can also be edited thanks to the new fields introduced. (#4682)
- A button "Discover Wikibase instances…" was added on the dialog which lists the registered Wikibase instances (#5007), whose design was improved (#5009)
- In the Wikibase schema editor, statements with non-standard datatypes (such as EDTF dates or musical notations) are now supported, assuming they use strings as underlying representation (#3263)
- The Wikibase issues tab now makes it possible to locate which rows are responsible for certain issues, using facets (#5033)
- The default throttle delay for the "Add column by fetching URLs" operation was reduced to 500ms and the error reporting for this field was improved (#5188)
- Wikibase templates (incomplete Wikibase schemas) can be saved and shared, as a way of helping contributors use the same way of structuring data in a Wikibase instance (#5043, #5303)
- The line-based importer now supports a custom delimiter, instead of only newlines (#4103)
- The Excel importer can be configured to import all cells as text, disabling the use of other datatypes supported by OpenRefine (#4838)
- The "some value" and "no value" Wikibase values can now be uploaded by OpenRefine (#5360)
- The Excel importer will also avoid coercing cell values to OpenRefine datatypes which do not fully fit them, such as representing a date as a date with time (#5389, #5390).
GREL changes
- Improved error handling in number formatting with the GREL
toString
function (#816) - The behaviour of the GREL function
wholeText()
has changed slightly in the way it handles newlines, following an upstream change in the jsoup library (jsoup issue #1636) - A new
parent
GREL function, to obtain the parent element of an XML element, was added (#5176)
Bug fixes
- The layout of the dialog to select a reconciliation match was improved so that the auto-complete widget does not hide the other options (#4821)
- Better in-tool documentation around the way the scatterplot facet detects numerical columns (#4890)
- The detection of URLs in cell values was fixed (#4546)
- The error message displayed when trying to add a Wikibase manifest with a manifest version that is too old or recent was improved (#4847)
- Errors returned by Jython expressions are more readable (#3012)
- The ODS exporter no longer creates a default sheet "Sheet1" in the documents it creates (#4864)
- Longer descriptions in auto-completion widget are not cut off anymore (#4988)
- The interface for editing cell values was improved to better explain how to input dates (#3082)
- The Windows
refine.bat
script was made more consistent with the Unixrefine
script (#4949, #5404) - The "Search for Match" dialog was rearranged so that the dropdown does not cover the buttons (#4945)
- Error handling in the scatterplot facet was improved (#4893)
- The "Collapse consecutive whitespaces" operation now handles unicode whitespace correctly (#4898)
- (#4991)
- The handling of GZIP-compressed files without
.gz
extension was improved in the importing pipeline (#547) - (#5153)
- The "Add column based on this column dialog" can be submitted by pressing "Enter" in the column field (#5143)
- The editing of redirected Wikibase items was fixed (#5162)
- The user experience was improved in the case of incomplete Wikibase schemas (#5131)
- The memory usage display was improved to show the used memory instead of the total memory, and was made more precise (#5222)
- The association of labels to form inputs was improved, enhancing the accessibility of the interface (#5239, #5242, #5249, #5284)
- An overflow issue in the reconciliation dialog was fixed (#5285)
- The Wikibase manifests now properly support locally-running reconciliation endpoints (#5035)
- The aspect ratio of Wikibase logos is now properly preserved (#5306)
- The SQL exporter interface was improved (#5224)
- The cell edit popup and dialogs with textbox inputs became resizable (#5330)
- When marking a set of cells as "New" in an unreconciled column, the user is prompted for the reconciliation service to use (#4985)
- More quality assurance checks were introduced in the Wikibase extension, such as checking for identical label and description in new Wikibase items (#4980)
- The caching of auto-completion results in the Wikibase extension was fixed (#5190)
- The Wikidata extension was fully renamed to "Wikibase extension" (#4525)
- The controls of the cluster and edit dialog are greyed out while clustering is taking place (#5369)
- The handling of unicode whitespace was improved throughout the application (#5105)
- Our MacOS packages (.DMG) are now properly signed and notarized, which should make their installation easier (#4586). Also, the presentation of the DMG image was made more user-friendly by including the customary link to the Applications folder. (#5509)
- The parsing of the unary minus sign in GREL was fixed (#5465)
- (From 3.7-beta3 on) The clustering dialog no longer introduces non-breaking spaces when selecting options with spaces in them (#5581)
- (From 3.7.1 on) The display of the memory usage during project import was fixed (#5665)
- (From 3.7.2 on) The localization in German was fixed (#5750)
- (From 3.7.3 on) Starting openrefine.exe on Windows with Java 17 was fixed (#5583)
- (From 3.7.3 on) Wikibase edits on deleted items are skipped and do not stall the entire batch (#5385)
- (From 3.7.3 on) The HTML document language is aligned with the language of the interface (#5925)
- (From 3.7.3 on) The default reconciliation types are displayed with both name and id (#5907)
- (From 3.7.3 on) The transpose cells across columns was fixed so it treats blank cells as null (#5229)
- (From 3.7.6 on) A missing space was added in the layout of the text facet (#6071)
- (From 3.7.6 on) Browser launching on startup was fixed for Snap-packaged OpenRefine (#6065)
- (From 3.7.6 on) The selection of Wikibase statement merging strategies was fixed (#6066)
- (From 3.7.6 on) A resizing issue in the presence of the Wikibase extension and facets was fixed (#6070)
- (From 3.7.7 on) A rendering issue in the Wikibase schema editor was fixed (#6165)
- (From 3.7.7 on) Attempts to fetch invalid URLs in the "Add column by fetching URLs" operation are properly reported to the user (#6141)
- (From 3.7.7 on) A missing space between the "remove" and "configure" buttons of the "Add column from reconciled values" operation was added (#6151)
For developers
- Some column metadata fields, which had never been exposed in the UI, have been removed (#5276)
- A new extension point to customize cell rendering was introduced (#5154)
Version 3.6
Starting with version 3.6, OpenRefine requires Java 11 or later.
New features
- The user is now warned when applying the "Fill down" or "Blank down" operations with a pending sorting criterion (#3256)
- The import preview refreshing can be disabled (#4009)
- Menu items to reveal collapsed columns were added to the column menus (#4067)
- The path to the
refine.ini
configuration file can now be changed on the command line (#4113) - It is now possible to download the JSON representation of the operation history, without resorting to copy and paste (#4498)
- It is now possible to work with Wikibase instances with federation enabled (#4287)
- The merge strategy for statements can be configured in the Wikibase schema editor. This also adds support for deleting statements. Beware that schemas created with earlier versions of OpenRefine will still use the original merge strategy. (#3383, #2116, #4130)
- OpenRefine can edit MediaInfo entities and not just Items (#4270)
- It is possible to disable the new version notification by setting the configuration variable
refine.display.new.version.notice=false
(#4410) - The dialog to reorder and delete columns was improved to easily delete most columns (#4557)
- The maximum editing speed and the Wikibase tag to apply to all Wikibase edits is now configurable for each Wikibase instance via its manifest (#3359)
- Extra URL fields in the starting page can be removed thanks to a new button (#4606)
- The "Use values as identifiers" operation now warns that it does not validate the identifiers (#3172)
GREL changes
- A new GREL function,
parent
, was introduced to obtain the parent element of an XML element (#4181) - A new GREL function,
scriptText
, was introduced to obtain the text contained in a<script>
or<style>
element in HTML (#4189) - The
random
(previouslyrandomNumber
) GREL function was improved (#3143) - A new GREL function
parseUri
was introduced (#1857) - A new GREL function
detectLanguage
was introduced (#642) - New GREL functions
encode
anddecode
were introduced (#148) - The error handling of the
pow
andexp
functions was improved (#3062) - The division operator returns NaN when computing 0/0 (#377)
- A function
timeSinceUnixEpochToDate
was introduced, to convert a duration since Epoch to a date object (#608) - A function
replaceEach
was introduced, to replace multiple substrings in one go (#2606)
Bug fixes
- XML and JSON importers now correctly create File and Archive columns when requested (#3462)
- Casing in UI text in English was made more consistent (#3363)
- Zip files with an unsupported compression method are correctly rejected by the UI at import time (#4112)
- Date parsing for Excel and ODS importers was fixed to avoid off-by-one errors due to changes in Daylight Saving Time (#4294)
- Labels of checkboxes in text filters are correctly bound to the corresponding checkboxes (#4243)
- The templating exporter correctly reports when a template contains errors (#3382)
- The dialog to add a Wikibase instance gives better error messages when the manifest's JSON structure is invalid (#4324)
- The missing separators when using the templating exporter in records mode have been fixed (#3955)
- The handling of curly braces inside export templates has been fixed (#3381)
- The options to parse column names in the CSV/TSV exporter have been turned into radio buttons (#4299)
- The project metadata dialog was improved to avoid being cut off when containing large values (#4119)
- The tabular exporter dialog was fixed to respect column order changes (#4271)
- The duplicates facet was fixed, it now works for all column names (#4233)
- Links from the tool to the documentation were improved (#3578)
- The XML importer correctly populates project metadata with import options (#4566)
- When running OpenRefine on Windows without Java, the user is now redirected to our docs about setting up Java (#4511)
- The alignment of OK/Cancel buttons was normalized to the right-hand side of all dialogs (#4564)
- The sizing of the SQL exporter dialog was fixed (#4579)
- Menu items which open a dialog end with a text ellipsis (#4585)
- Renaming a column is now done via a proper dialog and not a Javascript prompt (#4590)
- When creating a project by downloading the content of one or more URLs, all URLs are validated in the UI (#2687)
- A tooltip falsely indicating that project tags could be edited was removed (#3979)
- The spacing between the OK and Cancel buttons on some dialogs was fixed (#4622)
- Spell-checking was disabled on many text inputs where it does not make sense (#3004)
- The style of forms in the UI was made more consistent (#4639)
- The borders of dialogs was made uniform across the tool (#4662)
- The SQL exporter avoids adding invalid characters in its column names (#1940)
- The interface remembers whether the last reconciliation applied to a single cell or to all similar cells (#2316)
- Encoding issues in operation notifications have been fixed (#4540)
- The parsing of empty ODS files was fixed (#2465)
- The documentation around clustering was improved (#3033)
- Proxy settings are now respected by reconciliation operations (#3919)
- Encoding issues in project metadata fields were fixed (#3919)
- The dialog to select a Wikibase instance was improved (#3824)
- A bug causing workspace corruption when saving projects was fixed (#3719, #3277, #1418)
- The positioning of dialogs was improved (#4906)
- (from 3.6-rc1 on) The checks for Java versions in our start scripts have been updated to reflect that we support Java 11 to 17. (#4930)
- (from 3.6-rc1 on) The "Collapse consecutive whitespace" operation correctly recognizes all Unicode whitespace as spaces (#4883)
- (from 3.6-rc1 on) Menu items with ellipsis were fixed (#5003)
- (from 3.6.1 on) The editing of redirected Wikibase entities was fixed (#5162)
- (from 3.6.1 on) A bug with selection of clusters in the clustering dialog was fixed (#5138)
- (from 3.6.1 on) Date handling in the Google data extension was fixed (#5107)
- (from 3.6.1 on) A packaging issue in MacOS was tentatively fixed (#5160)
- (from 3.6.2 on) An overflow issue with the reconciliation dialog was fixed (#5286)
For extension developers
- (from 3.6-rc1 on) We migrated to jQuery 3.6.0. If you are using jQuery in your extension, some jQuery syntaxes that have been deprecated earlier might have been removed. If your extension runs with OpenRefine 3.5.2, you can check the web developer console for warning messages when the extension is used: fixing those should be enough for your extension to be compatible with OpenRefine 3.6 (#4891)
Version 3.5
New in 3.5.1
- Log4j was upgraded to version 2.16.0
- OpenRefine is compatible with Java versions 8 to 17 (#4106)