I tested running a bare-bone, Swift-based app on an iPod Touch (3rd gen) device. It appears Swift-based apps don't work with iOS 5.x but do work with iOS 6.x.
Here's what shows up in the debug log when I tried to launch the test app with iOS 5.0.1:
dyld: F_ADDFILESIGS failed for /private/var/mobile/Applications/59E31E79-9525-43B0-9DF6-8FEF3C0080F1/SwiftTestApp.app/Frameworks/libswift_stdlib_core.dylib with errno=1
dyld: F_ADDFILESIGS failed for /private/var/mobile/Applications/59E31E79-9525-43B0-9DF6-8FEF3C0080F1/SwiftTestApp.app/Frameworks/libswiftCoreGraphics.dylib with errno=1
dyld: F_ADDFILESIGS failed for /private/var/mobile/Applications/59E31E79-9525-43B0-9DF6-8FEF3C0080F1/SwiftTestApp.app/Frameworks/libswiftDarwin.dylib with errno=1
dyld: F_ADDFILESIGS failed for /private/var/mobile/Applications/59E31E79-9525-43B0-9DF6-8FEF3C0080F1/SwiftTestApp.app/Frameworks/libswiftDispatch.dylib with errno=1
dyld: F_ADDFILESIGS failed for /private/var/mobile/Applications/59E31E79-9525-43B0-9DF6-8FEF3C0080F1/SwiftTestApp.app/Frameworks/libswiftFoundation.dylib with errno=1
dyld: F_ADDFILESIGS failed for /private/var/mobile/Applications/59E31E79-9525-43B0-9DF6-8FEF3C0080F1/SwiftTestApp.app/Frameworks/libswiftObjectiveC.dylib with errno=1
dyld: F_ADDFILESIGS failed for /private/var/mobile/Applications/59E31E79-9525-43B0-9DF6-8FEF3C0080F1/SwiftTestApp.app/Frameworks/libswiftUIKit.dylib with errno=1
dyld: Symbol not found: _OBJC_CLASS_$_NSObject
Referenced from: /private/var/mobile/Applications/59E31E79-9525-43B0-9DF6-8FEF3C0080F1/SwiftTestApp.app/Frameworks/libswift_stdlib_core.dylib
Expected in: /usr/lib/libobjc.A.dylib
in /private/var/mobile/Applications/59E31E79-9525-43B0-9DF6-8FEF3C0080F1/SwiftTestApp.app/Frameworks/libswift_stdlib_core.dylib
For iOS 6.1.6, the app runs fine without displaying those error messages.